The Long Tail

Long Tail 101

[Note: I've updated my About page, which is where I've got the most concise definition of The Long Tail. This post is just a cleaned-up and easily linkable version of that.]

    The theory of the Long Tail is that our culture and economy is increasingly shifting away from a focus on a relatively small number of "hits" (mainstream products and markets) at the head of the demand curve and toward a huge number of niches in the tail. As the costs of production and distribution fall, especially online, there is now less need to lump products and consumers into one-size-fits-all containers. In an era without the constraints of physical shelf space and other bottlenecks of distribution, narrowly-target goods and services can be as economically attractive as mainstream fare.

    One example of this is the theory's prediction that demand for products not available in traditional bricks and mortar stores is potentially as big as for those that are. But the same is true for video not available on broadcast TV on any given day, and songs not played on radio. In other words, the potential aggregate size of the many small markets in goods that don't individually sell well enough for traditional retail and broadcast distribution may rival that of the existing large market in goods that do cross that economic bar.

    The term refers specifically to the yellow part of the sales chart above, which shows a standard demand curve that could apply to any industry, from entertainment to hard goods. The red part of the curve is the hits, which have dominated our markets and culture for most of the last century. The yellow part is the non-hits, or niches, which is where the new growth is coming from now and in the future.

    Traditional retail economics dictate that stores only stock the likely hits, because shelf space is expensive. But online retailers (from Amazon to iTunes) can stock virtually everything, and the number of available niche products outnumber the hits by several orders of magnitude. Those millions of niches are the Long Tail, which had been largely neglected until recently in favor of the Short Head of hits.   

    When consumers are offered infinite choice, the true shape of demand is revealed. And it turns out to be less hit-centric than we thought. People gravitate towards niches because they satisfy narrow interests better, and in one aspect of our life or another we all have some narrow interest (whether we think of it that way or not).

    Our research project has attempted to quantify the Long Tail in three ways, comparing data from online and offline retailers in music, movies, and books.

1. What's the size of the Long Tail (defined as inventory typically not available offline?
2. How does the availability of so many niche products change the shape of demand? Does it shift it away from hits?
3. What tools and techniques drive that shift, and which are most effective?

    The Long Tail article (and the forthcoming book) is about the big-picture consequence of this: how our economy and culture is shifting from mass markets to million of niches. It chronicles the effect of the technologies that have made it easier for consumers to find and buy niche products, thanks to the "infinite shelf-space effect"--the new distribution mechanisms, from digital downloading to peer-to-peer markets, that break through the bottlenecks of broadcast and traditional bricks and mortar retail. 

    The Wikipedia entry on the Long Tail does an excellent job of expanding on this.

It's not about rags to riches

Alessio Iacona, an Italian journalist, has a question:

Dear Mr Anderson,

Recently, I've had the chance to write more than once about your interesting theory called "The Long Tail". But now I'd like to ask you to comment on a brand new Long Tail case.

In 1995 a quite anonymous chef named Simon Hopkins published a book called  "Roast Chicken and Other Stories". For about ten years the book lived an obscure life on library bookshelves until a panel of leading chefs, writers and restaurateurs nominated it "the most useful cookery book of all time". Then the "Long Tail experience" took place: a lot of people went out to buy the book but they couldn't find it in bookshops. So they turned to Amazon.co.uk, which received so many orders that "Roast Chicken and Other Stories" jumped in 48 hours on the first place of the 100 top sellers list, passing the sixth volume of Harry Potter.

My question: What do you think about it?

Alessio,

    Thanks for the example, which is an interesting one. It's not really a Long Tail case, however, because the LT in books doesn't really start until about Amazon rank 100,000 (prior to the nomination that book had been in the realtively-healthy 40,000s) and I imagine the title was actually still available in most UK bookstores. But it does show the virtue of keeping the back catalog available.

    Although I used a similar example ("Touching the Void") to begin my Wired article last year, such rags-to-riches tales are not really the typical LT story. The point I'm making in my writing is that products don't have to become hits to be a success and fill a need. Now that online retail economics allow us to offer and sell modest-selling niche products efficiently, we don't have to settle for one-size-fits-all products of broad appeal. Narrow appeal is often better when it's narrowly-focused on your interests, and today the economics of digital distribution allow such micro-markets to exist profitably alongside the mainstream.

    The fact that the occasional title is catapulted from the Long Tail to the Short Head is certainly a reminder that great stuff can be found in the tail, but it's the exception rather than the rule. A much more frequent, and better, example is the niches in the tail that are finding new demand, two, three, even ten times what they'd seen before thanks to the power of the Internet to help people find and obtain non-mainstream fare. That doesn't make them hits; it just makes them more successful niches. But that's the real power of the Long Tail.

Best,

Chris

Do you really need both ends of the curve?

I've argued, again and again, that successful Long Tail aggregators have to have both a head (hits) and a tail (niches) to work well. It's not enough to just have the tail. People need to start with what they already know and then discover, via effective recommendations and other filters, niches that suit them better.

    But Adina Levin of BookBlog says I'm partly wrong. And, alarmingly, I find myself agreeing with her. She argues:

LiveJournal and Flickr disprove his theory. LiveJournal is an online journal community that has historically had a large population of young people. They congregate in social groups, often starting with people who are friends offline...Similarly, Flicker is an online photo sharing community, where users can share photos with their friends and the world.

Cultural preferences are social. When people like strange music, unusual fashions, or minority religious practices, they most often do so with a subculture of like-minded folk.

Until now, the smaller social networks in which people share culture have been largely private and noncommercial, with a small number of exceptions, like Tupperware parties and Amway.

So, the successful examples of social content-sharing are [also] based on non-commercial content, like LiveJournal and Flickr. There are also grassroots networks of cross-linked music blogs where people review and recommend music. And there are networks of cross-linked knitting blogs where people review and recommend patterns. Classic long-tail stuff.

Chris Anderson is right that catalog retailers like Netflix and Amazon need to have hits, which help draw users to the niche. Their recommendation engines serve as an automated proxy for the natural social recommendations that people make every day.

But that's true only when you start with the content. When you start with groups of people, then opportunities for "long tail" are abundant, and don't depend quite so much on mainstream content.

    Busted. I'm guilty of two sins. First, as she says, putting content before community. True enough. But are there any simple principles that can explain why some of those community sites grow and sustain, while most flit into and out of existence at the speed of a teenage attention span?

    I'd argue that what successful community sites like LiveJournal, Flickr and MySpace have is content on top of community. There's something to do and see there, above and beyond simply socializing. They have something the Friendsters of the world, like the SixDegrees before them, do not. In short, Long Tail content creates a stickiness that can take a social site beyond the lifespan of a fad.

    I'm also guilty of considering Long Tail content aggregators mostly from the perspective of mainstream content, such as music and movies, where there are both hits and niches. But in the user-created world, such as photos and blogs, it's often just varying degrees of niche. There is no head to go with that tail.

    Which raises an interesting question: We know that community sites can become user-created content aggregators. But is the reverse true, too? Are there any good examples of user-created content aggregators that succeeded without significant community?

    I can think of a few examples, but none strong enough to suggest a rule. Technorati is a blog aggregator without a community, but I'd agree with Steve Rubel that its success is largely due to filling a need that will soon be considered a standard feature of any good search engine. Bloglines is more of a tool to read feeds from blogs you've found some other way, rather than a true aggregator. There are probably aggregators in every peer-production niche you can think of, from fan fiction to cross-stitch patterns, but I'll bet most of the successful ones have a strong community aspect. Hmmm. I'll continue to ponder this one while hoping that readers smarter than me can answer it.

Filters 101

There's been some interesting discussion in the comments about what kind of filtering works best and under which conditions. Reading it, it occurred to me that I haven't seen a good taxonomy of filter types, which would probably  help tremendously. (Yes, I know, taxonomies are so last century; feel free to collectively remix this into a much more modern folksonomy out there on your own blogs). I'll lay out a first pass at a family tree here and then revise based on feedback.

There are, as I see it, two main categories of filters (or, to be precise, "post-filters")--Software and People--with several subtypes and loads of different variation in the wild. In the following construction, I'm probably missing some sub-species (or an entire genus or two), so please offer suggestions and corrections in the comments.

The Two Families of Filters:

• Software (recommendations)
  
  • Wisdom of crowds (algorithms that measure collective opinion or behavior)
    • Buzz: Google PageRank, del.icio.us, Technorati
    • Ratings (active*): Netflix recommendations, eBay ratings
      
    • Behavior (passive**): Amazon recommendations, Audioscrobbler, Yahoo Music custom radio stations
      
  • AI: Semantic analysis, audio analysis
    
   
• People (tastemakers, influentials)
  
  • Pros (editors): Critics, celebrities, Rhapsody category editors, librarians, SavageBeast (Pandora) musicologists (+)
  • Amateurs (mavens): Blogs, playlists, customer reviews, Wikipedia contributors, smart friends (++)
    
  • Mobs (distributed intelligence): taggers, linkers, file-traders (+++) 

-------
Notes:

* "Active" filtering: These are systems that ask you if you like or dislike things, so they can offer you better choices in the future. Because they require users to make an effort to participate, typically by clicking on some rating system, they're vulnerable to sampling bias. That can range from who participates to how they choose to interpret their choices, be it thumbs up and thumbs down or one to five stars..

** "Passive" filtering: Also known as collaborative filtering, this is software that tracks the behavior and actions of lots of people and extracts meaning and recommendations from that. Amazon's "people like you" recommendations are the best-known example, but hundreds of others services, from Netflix to Yahoo, do the same. The advantage of passive filtering is that it's based on what people do, not what they say. As such, it tends to avoid the participation bias mentioned above.

---

+  By "pros", I mean people typically working within some service to help guide consumer, such as Rhaposdy editors, although they could also be independent domain experts who have built a business around their influence.

++ By "amateurs", I mean power consumers, people who are passionate and informed enough about a category to advise their peers.

+++ People who are part of a "mob" may not think of themselves as offering recommendations or guidance at all. They're just doing what they do. The fact that other people are following their example is a consequence of network effects at work, not necessarily the intention of the members.

How finely can you slice aggregation?

As I mentioned in the previous post, the three main Long Tail business opportunities are:

1. Long Tail aggregators (that include both the head and tail of content and products)
2. Niche suppliers/producers (who get aggregated by someone else)
3. Filters (which help people find what they want)

    Most of the examples I've been using to date, such as Netflix, Amazon and iTunes, fall into the first of these categories, aggregation. But as the smart kids in the front row always point out, there's a seeming paradox at work in that category.

    The Long Tail is all about the shift from hits to niches. But aren't all those aggregators "hits"? They're not only the largest players in their category, but they seem to be getting even larger, gaining market share at the expense of their competitors. Is there something about aggregators that tends to favor a few big winners, even as the other two categories fragment into a million niches of varying size?

    That's certainly the way it looks now, but I suspect it won't last, or at least won't last as it is. I'll give an example by starting with the biggest of them all, Google.

    In a sense, Google is a classic Long Tail aggregator, one that uses great filtering (its PageRank algorithms) to improve the s/n ratio of the tail. And not just one tail. They aggregate the tail of information, the tail of advertisers and the tail of publishers (see this post for more). But is plain old Google enough for all of this? Or is there something more appropriate for many needs than one-size-fits-all search?

    The question can be generalized to this: how finely can you slice aggregation? The answer, in the case of search, turns out to be quite finely indeed. The rise of the "vertical search" market is simply a case of slicing aggregation into niches, optimized for different needs. (Actual examples here; imaginary ones here)

    In fact, Google itself is already doing this, with Google Local, Google Scholar, Google Maps, Froogle, Google News, Google Print, and so on. Each has a specialized presentation and pulls from a subset of the information universe that gives more appropriate and useful results in that domain. Right now that's Google fine-slicing its own aggregation--the emergence of a long tail of search, but within a short head of search companies--but there are plenty of other firms that are rushing into this market to do the same.

    Now let's apply the same approach to media and entertainment aggregation. I've already argued that any Long Tail aggregator has to include both the head and tail of content, to allow the recommendations and filters to carry users from the known to the unknown. But does that only work in broad categories, such as music and film, or could it work just as well, if not better, at the sub-genre level, aggregating the head and tail of just jazz, for instance, or documentaries?

The advantages would be that, instead of a one-size-fits-all presentational model, such as is currently the case in iTunes, you could have a model customized for a genre. I don't know much about jazz, but I'll bet that it might be useful for a jazz music service to have links for each one of the musicians on an album (because they often shift from group to group), much as IMDB tracks the careers of each movie actor and director independently. But that wouldn't be appropriate for pop music, because the individual musicians aren't as important.

    Likewise for documentaries. Wouldn't it be useful to have documentaries accompanied by related information and news articles, so the films could be a launching-off point for further exploration of that subject? Or a site just for TV-show DVDs, complete with all the entertainment news and gossip around their stars?

    No doubt there are already companies working on all of these and many more. What's important, however, is that these fine-slice aggregators allow themselves, in turn, to be aggregated by the larger, more general aggregators. Those niche music services should, for starters, be fully searchable by Google and anyone else. Could they also do deals with iTunes to be niche specialists, the way individual merchants are integrated into Amazon's Marketplace and eBay's Stores? I'll bet they could. It's the Long Tail of aggregation, coming soon.

Does the Long Tail apply outside of media and entertainment?

My original article and most of the examples I've focused on here are in the media and entertainment industries, mostly because that's where the most accessible examples are. Music, video, and books are fun, universally understood, and have the best statistical evidence of the Long Tail at work. There's just a ton of data there. Ten years of Amazon, five years of Netflix and Rhapsody and at least that long for all their kin have shown just how dramatic the effect of infinite shelf space can be in revealing the true shape of consumer demand.

    But lest you think that's the extent of it, let me hasten to emphasize that the Long Tail goes far beyond media and entertainment. It extends from advertising to hard goods, from services and software to people.

Q: So why don't you write more about all that here?
A: I'm saving it for the book, which will be at least a third about the Long Tail in physical goods, services, advertising and so on.

Q: Why?
A: It's a very reasonable deal I worked out with my publisher, Hyperion, which has been very open-minded about this experiment in book writing in public. Knowing that there will be a big chunk of fresh, and I hope equally interesting, research and analysis of the Long Tail in industries and markets where one might not thought it would be a force should be a good reason to buy the book, even for those who are already fluent entertainment longtailers.

Q: Fair enough. But are there any broad differences between the media and entertainment industries and the others, such as physical goods, that are worth keeping in mind?
A: Yes. As the graphic above (click on it for a readable version), which is taken from the original article, implies, there are two steps to the Long Tail. The first is digital catalogs of physical goods, such as those of Amazon or eBay. These have unlimited capacity and can offer search, recommendations, infinite information about products and flexible categorization. That improves the economics of retail dramatically compared to traditional stores, which allows them to carry far more products. This gets you half-way down the Tail

    But to get all the way down the Tail you need digital catalogs of digital goods. Think everything from iTunes to Google. This is where the economics are best of all: near-zero marginal costs of manufacturing and distribution. What that means is that you can basically stop discriminating entirely, and give up trying to guess what people want. Offer everything and let the marketplace decide what it wants. Chances are you'll be surprised by what people gravitate to, not just in the unexpected demand for niches but in the immense range of tastes. That's the Long Tail.

Why is this blog so, well, wonky?

When the supersmart members of the Stanford MBA team that's helping with the book research don't quite follow my posts, I know I've gone down the rabbit hole of obscurity.  (I think it was the signal/noise debate that did it).  But their polite bafflement was a useful reminder not to get too abstract here. It also raises the question of what level to pitch the discussion in the Long Tail book.

Should the book be series of gripping narrative stories around a phenomena, like The Tipping Point? (If only!) A collection of historical examples and academic research to illustrate an overarching theory, like The Wisdom of Crowds?  Or a relatively straightforward, data and anecdote-fueled articulation of a big thesis, like Out of Control? I'm leaning towards the last of those, mostly because that's what comes most naturally and fits with where my research is going.

I wish I were as good a storyteller as Malcolm Gladwell and that my tales were as narratively rich. But I'm not, and they're not. Instead, what I hope to offer is clarity, via data and a decent grounding in economics, lightened with loads of fun case studies and telling anecdotes.

In the meantime, a slight explanation for why I've been indulging in so much theory here. I originally trained to be a physicist, in part because my hero growing up was Richard Feynman. One of the virtues of physics is that it's based on the concept of understanding the world via first principles, the underlying rules that explain all the complexity around us.

Before we came up with the chemical table of the elements and understood the underlying rules of valence theory and atomic structure, the properties of matter were impossible to understand. Likewise for the movement of the planets and stars before Newton's Laws. Today biology remains brain-meltingly complicated largely because we don't have a good underlying theory for it.

What I'm trying to do here is to establish the first-principle rules of the Long Tail. I realize that the search for a grand unified theory is usually a recipe for ending your days muttering at a blackboard covered in scribbles. But I do think that the economics of abundance are poorly understood, and the Long Tail is as good an opportunity as any to lay out some pointers to how they might work. With your help, we'll work through some of that here and I'll find a way to make it easier to digest in the book.

What the Long Tail isn't

After witnessing much misuse of the Long Tail phrase, this silly post has finally pushed me over the edge.  It's time to draw the line. Long Tails are found everywhere, but not, you know, actually everywhere.

There are many distortions of the term, but the most common one is to use it as a newly-positive synonym for "fringe". Invoking the Long Tail is not a magic wand to explain away the apparent lack of demand for what you've got.  The Long Tail is not a get-out-of-jail-free card for poor-selling product. Or weak sectors. Or bad ideas.

The fact that something isn't popular doesn't mean that it's just a matter of time before it will benefit from all sorts of  powerful demand-creation Long Tail effects. More likely, it's just not good enough to be commercially interesting, and probably never will be.

Most of the "niche" products in the tail are simply crap. That's okay, because some significant fraction of them aren't and with a functioning way to separate the good from the bad, they can be found by those who will appreciate them. But note the essential element: a functioning way to drive demand.

As I've mentioned in the original article, for Long Tail effects to work, you need both a head of relatively few hits and a tail of many niches, so that recommendations and other filters can lead consumers from one to the other.

A tail without a head is too noisy and apparently random to get consumer traction; people need to start with the familiar and then move, via trusted recommendations, to the unfamiliar. Likewise a head without a tail is too limited in choice; the odds of finding a niche you want are too low to bother exploring much beyond what you already know.

Thus the two big Long Tail opportunities are:

1. Aggregating hits and niches into a one big curve, from head to tail.
2. Creating content and products that can plug into someone else's aggregated curve.

If what you're selling is fringe, it may well enjoy Long Tail benefits, but only if it can fit nicely into an existing market that has the capacity to drive demand. If that market doesn't exist, it's unlikely that throwing some niche products out there is going to create it. Even if it does exist, those products will only reach their audience if the filters and recommendations are good enough to find them.

While I'm at it, here are some other things the Long Tail is not (to protect the guilty I will refrain from linking to the offending examples):

• Commodification
  • The LT is about nicheification, which is different.
• Simple variety
  • Offering a few different choices or a bit of customization (like the sandwich filling options in the risible example above) is not enough. Long Tail effects kick in when you're expanding variety and choice by orders of magnitude--from 10x to infinity.
• The case for an all amateur, self-published future
  
  • The LT will probably have as much commercial content as ever. It will just be joined by far more amateur fare, forming a relatively seamless continuum from pros to ams.
• The actual end of hits
  • The LT ends the tyranny of hits, shifting the market equally to niches. But it certainly acknowledges that some things will continue to be a lot more popular than others. Powerlaw distributions are as natural as diversity itself.
• A focus on small markets at the exclusion of large ones
  • Again, you need both hits and niches to allow the filters and recommendation engines to work, driving demand down the curve from the known to the unknown.
• Just any powerlaw
  • Powerlaws are ubiquitous. Long Tails are not. The first shows up anywhere you have variety, inequality, and network effects (word of mouth). The second requires massive variety and a wide range between the hits and niches. After all, many short tails are simply truncated powerlaw distributions. They just aren't, er, long.

Venture capitalists who are tired of bogus Long Tail pitches are invited to point the hapless entrepreneurs to this post.

Is the Long Tail full of crap?

One of the most frequent mistakes people make about the Long Tail is to assume that things that don't sell well are "not as good" as things that do sell well. Or, to put it another way, they assume that the Long Tail is full of crap. After all, if that album/book/film/whatever were excellent, it would be a hit, right?

Well, in a word, no. Niches operate by different economics than the mainstream. And the reason for that helps explain why so much about Long Tail content is counterintuitive, especially when we're used to scarcity thinking.

First, let's get one thing straight: the Long Tail is indeed full of crap. But it's also full of works of refined brilliance and depth--and an awful lot in between. Exactly the same can be said of the Web itself. Ten years ago, people complained that there was a lot of junk on the Internet and, sure enough, any casual surf quick