The Long Tail

It's not about rags to riches

Alessio Iacona, an Italian journalist, has a question:

Dear Mr Anderson,

Recently, I've had the chance to write more than once about your interesting theory called "The Long Tail". But now I'd like to ask you to comment on a brand new Long Tail case.

In 1995 a quite anonymous chef named Simon Hopkins published a book called  "Roast Chicken and Other Stories". For about ten years the book lived an obscure life on library bookshelves until a panel of leading chefs, writers and restaurateurs nominated it "the most useful cookery book of all time". Then the "Long Tail experience" took place: a lot of people went out to buy the book but they couldn't find it in bookshops. So they turned to Amazon.co.uk, which received so many orders that "Roast Chicken and Other Stories" jumped in 48 hours on the first place of the 100 top sellers list, passing the sixth volume of Harry Potter.

My question: What do you think about it?

Alessio,

    Thanks for the example, which is an interesting one. It's not really a Long Tail case, however, because the LT in books doesn't really start until about Amazon rank 100,000 (prior to the nomination that book had been in the realtively-healthy 40,000s) and I imagine the title was actually still available in most UK bookstores. But it does show the virtue of keeping the back catalog available.

    Although I used a similar example ("Touching the Void") to begin my Wired article last year, such rags-to-riches tales are not really the typical LT story. The point I'm making in my writing is that products don't have to become hits to be a success and fill a need. Now that online retail economics allow us to offer and sell modest-selling niche products efficiently, we don't have to settle for one-size-fits-all products of broad appeal. Narrow appeal is often better when it's narrowly-focused on your interests, and today the economics of digital distribution allow such micro-markets to exist profitably alongside the mainstream.

    The fact that the occasional title is catapulted from the Long Tail to the Short Head is certainly a reminder that great stuff can be found in the tail, but it's the exception rather than the rule. A much more frequent, and better, example is the niches in the tail that are finding new demand, two, three, even ten times what they'd seen before thanks to the power of the Internet to help people find and obtain non-mainstream fare. That doesn't make them hits; it just makes them more successful niches. But that's the real power of the Long Tail.

Best,

Chris

Great Business Idea (Free to a good home)

While we're on the topic of music aggregators, it seems clear to me that more is always better when it comes to band writeups/background. I love the Rhapsody/iTunes/Napster/YahooMusic model of listening to music, but I almost always want more details on the bands. When I'm listening to a playlist and a song comes on that I like; I want to read more than a few sentences about them. But for anything other than the usual hit artists, that's rarely available. Which suggests a great business, waiting to be built:

    Wikipedia meets Rhapsody

    If anyone's motivated and equipped to flesh out a band's bio and other details, it's its fans (and band members). But unless I'm missing something, none of the major online music services allow this sort of user-contributed content (somebody please correct me if I'm wrong). We're stuck with whatever the service's own editors supply, or whatever one-size-fits-all metadata came with the files. It's rarely enough.

    Part of the reason this isn't offered may be that these services tend to use proprietary client software running locally on your own PC, so the usual web/wiki conventions of open participation may be tricky to implement. But in the age of Ajax, I don't see why that has to be the case. (No doubt DRM has something to do with it. But then why not a web/client combo, that uses a web interface for content and only uses the client for DRM/steaming?)

    A few services, such as MSN Music, which is web-based, allow user reviews, but no more than that. Why not let users write/edit the band bio, discography, tour dates, reviews, essays, influences, pictures, and everything else? It works for Wikipedia, after all; this would just be a fine-sliced version of that, focused only on music.

    Part of the amazing success of MySpace is that it's built on user-created content, including a load of band sites that are controlled by the bands themselves. Indeed, MySpace may be the best one to roll out such a new offering, combining a full-featured music stream/download service that has the same 1m tracks of all the others, along with the peer-production indie music and band sites that has made MySpace such a phenomena already.

    Whether in their hands or someone else's, this is an idea whose time has come. The users want more. And they're ready to help.

Niche aggregators: that was fast!

Just when I think I'm really getting way too theoretical here, the world suddenly proves that if anything I'm behind the times. Thanks to Mayhem and Chaos for pointing out that while I've been proposing the idea of fine-sliced (niche) music aggregators, Audiolunchbox (and, presumably, CD Baby soon) has been building them.

I finally realized that CD Baby is the perfect example of long tail -- a large chunk of the CDs at CD Baby sell only a few copies in their life-time, yet CD Baby makes them all available. The idea to create focused smaller stores that serve specific genres, are essentially tails within tails.

For instance, a micro store "louisianabeats.com" might focus on zydeco and other Louisiana centric music. Anyone who loves Zydeco and heads to CD Baby, might not be able to find the right CDs in the vast CD Baby catalog. Instead, CD Baby does the work of identifying all the zydeco music and focusing it on louisianabeats.com. This site in itself will have a long tail effect -- some CDs will sell much better than others. And because of this specific focus, these better selling CDs are getting another shot at becoming popular.

In the grand scheme of things, CD Baby is on the long tail side and Amazon is on the short head side. Inside of the long tail "louisianabeats.com" creates another tail, "ohiobeats.com" another and so forth. This could even go on to "calibeats.com", which in turn could have smaller "norcalbeats.com" and "socalbeats.com" tails within tails.

And to prove [CD Baby]  is totally on the right track, online MP3 retailer audiolunchbox.com today [July 20] announced the creation of "7 genre specific digital music stores": themdirtyblues.com, pureclassical.com, punktracks.com ...  

The others are CountryTracks.com, TheLoveOfMetal.com, DownloadPop.com and AudioFader.com, "which will encompass electronic, dance, house, trance and more." They're not up and running yet, but promise to launch this summer.

Exhibit [A] in the case for niche aggregators

Not sure where I was going with my "fine-slicing aggregation" post? Steve Rubel comes up with a perfect example to demonstrate:

As I browse through the podcasts on the iTunes Music Store I notice something really important is missing - The Long Tail. While each album listing points me to others people have purchased, the same metadata is missing when it comes to podcasts. We should be able to see a link that says "people who subscribe to this podcasts also subscribe to..." Take a look at the images below and note the differences.

The lesson: the best way to present one kind of product is not necessarily the best way to present another. I gave the example of hotlinking the individual performers on Jazz albums, much as IMDB does for actors and directors, so you can follow their individual careers. But the podcasting example is even better.

As this demonstrates, the iTunes interface is too inflexible (so far) to present podcasts as usefully as it presents pop music. That's either an opportunity for Apple to adapt iTunes so it offers different UIs for different kinds of content (much as Google has with everything from Google Maps to Google Print) or for a specialized aggregator focused just on podcasting to rise. Either way, this shows why one-size-fits-all aggregation doesn't work, and how to slice it finer to suit niches better.

Sidebar Shoutout II

Just a quick note to say that I've added another dozen entries to the "Long Tail comment elsewhere" sidebar. You can now subscribe to its feed independently (and more than a hundred of you have already), so if you're interested in tales of the tail from around the web, copy this link into your feedreader and I won't have to post many more of these reminders.

Do you really need both ends of the curve?

I've argued, again and again, that successful Long Tail aggregators have to have both a head (hits) and a tail (niches) to work well. It's not enough to just have the tail. People need to start with what they already know and then discover, via effective recommendations and other filters, niches that suit them better.

    But Adina Levin of BookBlog says I'm partly wrong. And, alarmingly, I find myself agreeing with her. She argues:

LiveJournal and Flickr disprove his theory. LiveJournal is an online journal community that has historically had a large population of young people. They congregate in social groups, often starting with people who are friends offline...Similarly, Flicker is an online photo sharing community, where users can share photos with their friends and the world.

Cultural preferences are social. When people like strange music, unusual fashions, or minority religious practices, they most often do so with a subculture of like-minded folk.

Until now, the smaller social networks in which people share culture have been largely private and noncommercial, with a small number of exceptions, like Tupperware parties and Amway.

So, the successful examples of social content-sharing are [also] based on non-commercial content, like LiveJournal and Flickr. There are also grassroots networks of cross-linked music blogs where people review and recommend music. And there are networks of cross-linked knitting blogs where people review and recommend patterns. Classic long-tail stuff.

Chris Anderson is right that catalog retailers like Netflix and Amazon need to have hits, which help draw users to the niche. Their recommendation engines serve as an automated proxy for the natural social recommendations that people make every day.

But that's true only when you start with the content. When you start with groups of people, then opportunities for "long tail" are abundant, and don't depend quite so much on mainstream content.

    Busted. I'm guilty of two sins. First, as she says, putting content before community. True enough. But are there any simple principles that can explain why some of those community sites grow and sustain, while most flit into and out of existence at the speed of a teenage attention span?

    I'd argue that what successful community sites like LiveJournal, Flickr and MySpace have is content on top of community. There's something to do and see there, above and beyond simply socializing. They have something the Friendsters of the world, like the SixDegrees before them, do not. In short, Long Tail content creates a stickiness that can take a social site beyond the lifespan of a fad.

    I'm also guilty of considering Long Tail content aggregators mostly from the perspective of mainstream content, such as music and movies, where there are both hits and niches. But in the user-created world, such as photos and blogs, it's often just varying degrees of niche. There is no head to go with that tail.

    Which raises an interesting question: We know that community sites can become user-created content aggregators. But is the reverse true, too? Are there any good examples of user-created content aggregators that succeeded without significant community?

    I can think of a few examples, but none strong enough to suggest a rule. Technorati is a blog aggregator without a community, but I'd agree with Steve Rubel that its success is largely due to filling a need that will soon be considered a standard feature of any good search engine. Bloglines is more of a tool to read feeds from blogs you've found some other way, rather than a true aggregator. There are probably aggregators in every peer-production niche you can think of, from fan fiction to cross-stitch patterns, but I'll bet most of the successful ones have a strong community aspect. Hmmm. I'll continue to ponder this one while hoping that readers smarter than me can answer it.

A 4th Long Tail business category: Tools

Danny Kim makes a good point in this comment:

I think that there is a fourth long tail business: tool-makers. For a long tail to have niche content producers, like amateur bloggers or podcasters, they need tools which will lower the barrier-to-entry to content-producing for those niche producers.

You could say long tails might be a natural byproduct of those already-existing tools, including gadgets like camera phones and mp3 recorders. But in reality, there's a good number of companies that are jumping into the business of making tools so that "we" can produce content easily. Some examples from the blogging world: Movable Type, Typepad, Blogger, Odeo (podcasting).

I think he's right. It also dovetails nicely with a point I've been making in speeches of late (and fleshing out in the first chapter of the book): there are three ascendant forces that are creating an era of Long Tails.

They are:

1. Democratizing the tools of production (example: the PC)
2. Lowering the transaction costs of consumption (example: the Internet)
   
3. Connecting consumers to drive demand to niches (example: filters)

The first extends the tail to the right, the second raises the volume of sales/use in the tail, and the third drives demand down the tail, from hits to niches. Like this:

In the past, I've correlated each of the three main Long Tail business categories to one of these forces:

    Force                                                  Business

1)  Democratize production                          Long Tail producers
2)  Lower the cost of consumption               Long Tail aggregators
3)  Connect consumers to drive demand       Long Tail filters

Now, although it's less neat, I should probably revise that to this:

    Force                                                 Business

1)  Democratize production                         Long Tail toolmakers, producers
2)  Lower the cost of consumption               Long Tail aggregators
3)  Connect consumers to drive demand      Long Tail filters

Thanks, Danny!

Filters 101

There's been some interesting discussion in the comments about what kind of filtering works best and under which conditions. Reading it, it occurred to me that I haven't seen a good taxonomy of filter types, which would probably  help tremendously. (Yes, I know, taxonomies are so last century; feel free to collectively remix this into a much more modern folksonomy out there on your own blogs). I'll lay out a first pass at a family tree here and then revise based on feedback.

There are, as I see it, two main categories of filters (or, to be precise, "post-filters")--Software and People--with several subtypes and loads of different variation in the wild. In the following construction, I'm probably missing some sub-species (or an entire genus or two), so please offer suggestions and corrections in the comments.

The Two Families of Filters:

• Software (recommendations)
  
  • Wisdom of crowds (algorithms that measure collective opinion or behavior)
    • Buzz: Google PageRank, del.icio.us, Technorati
    • Ratings (active*): Netflix recommendations, eBay ratings
      
    • Behavior (passive**): Amazon recommendations, Audioscrobbler, Yahoo Music custom radio stations
      
  • AI: Semantic analysis, audio analysis
    
   
• People (tastemakers, influentials)
  
  • Pros (editors): Critics, celebrities, Rhapsody category editors, librarians, SavageBeast (Pandora) musicologists (+)
  • Amateurs (mavens): Blogs, playlists, customer reviews, Wikipedia contributors, smart friends (++)
    
  • Mobs (distributed intelligence): taggers, linkers, file-traders (+++) 

-------
Notes:

* "Active" filtering: These are systems that ask you if you like or dislike things, so they can offer you better choices in the future. Because they require users to make an effort to participate, typically by clicking on some rating system, they're vulnerable to sampling bias. That can range from who participates to how they choose to interpret their choices, be it thumbs up and thumbs down or one to five stars..

** "Passive" filtering: Also known as collaborative filtering, this is software that tracks the behavior and actions of lots of people and extracts meaning and recommendations from that. Amazon's "people like you" recommendations are the best-known example, but hundreds of others services, from Netflix to Yahoo, do the same. The advantage of passive filtering is that it's based on what people do, not what they say. As such, it tends to avoid the participation bias mentioned above.

---

+  By "pros", I mean people typically working within some service to help guide consumer, such as Rhaposdy editors, although they could also be independent domain experts who have built a business around their influence.

++ By "amateurs", I mean power consumers, people who are passionate and informed enough about a category to advise their peers.

+++ People who are part of a "mob" may not think of themselves as offering recommendations or guidance at all. They're just doing what they do. The fact that other people are following their example is a consequence of network effects at work, not necessarily the intention of the members.

America's record store, like it or not

When I was a slacker twentysomething, I, like many of my slacker twentysomething kin, worked in a record store. It was a pretty big record store in the business downtown of Washington DC, part of a chain that no longer exists and whose name now escapes me. It catered mostly to the lawyers, admins and paralegals who worked around there, so it was pretty mainstream, but I still remember the aisle of import records (mostly British new wave, since it was the mid-80s) that stretched the length of the store near the stool where I sat watching the door and answering questions. The entire back wall was 12-inch singles, and classical had its own room, with excellent acoustics for refined listening.

    This all came back to me last week as I toured Wal-Mart's music department, doing research for the book. Wal-Mart accounts for about a fifth of America's music sales (I mistakenly said a quarter in this post), and is by far the nation's largest music retailer. As such, it was an essential field trip, mostly to learn more about the opposite of the Long Tail.

    Wal-Mart is the Short Head. And I was soon to discover just how short short really is.

    This is probably the right time to confess that until I visited I was probably the only person in the country who had never been to a Wal-Mart. It's not that I'm a snob; I'm just not much of a shopper and when I do the big box thing it's Costco. But some 138m Americans shop at Wal-Mart each week, making it perhaps the single most unifying cultural force in the country. So everyone else has already noticed what I discovered, which is that Wal-Mart is a charmless but impressive demonstration of the power of container ships and monopsony to arbitrage global labor rates. Everything is freakishly, how-do-they-do-that cheap.

    But it's a depressing way to buy music. Although the size of the inventory varies from store to store, the average number of titles in each, which was 5,000 last year (there are, as a point of reference, 800,000 CDs available on Amazon), has reportedly fallen since then as shelf space for music has been given over to DVDs. At the store I visited in Oakland, California there were about 4,100 titles, distributed as follows:

• "Rock/Pop/R&B":            1800
• "Latina":                           1500
• "Christian/Gospel":             360
• "Country":                          225
• "Classical/Easy Listening":  225

    There were two main aisles. One was "Rock/Pop/R&B"; the other was "Latina". All other categories were lumped into single racks, such as this one:

That is, by the way, the entire Jazz, Classical, World Music, Easy Listening and New Age section.

    Of the estimated 30,000 new albums released each year, Wal-Mart carries just 750, according to David Gottlieb, a former label executive interviewed in this Frontline documentary. Entire categories, from dance to spoken-word, are either missing or buried in Rock/Pop/R&B.

    [I should note here that this just applies to the physical stores. Walmart.com actually has what appears to be a pretty good music site, with 80,000 CDs, 500,000 downloadable tracks at a market-leading $0.88 each and the option to create your own mix CDs. Unfortunately it doesn't support Firefox, so I can't tell you much more than that.]

    Rolling Stone had a revealing article last year about the power of Wal-Mart in the music industry and the effect on our culture of its small (and shrinking) music shelf space and ban on music with a Parental Warning sticker.

No one in the music business ever expected Wal-Mart to become the most powerful force in record retailing. In the past, the business was shared among smaller local and regional chains such as Musicland, which once had an estimated ten percent of the market. But as Wal-Mart and other national discount operations such as Target and Best Buy have grown -- approximately half of all major-label music is sold through these three -- an estimated 1,200 record stores have closed in the past two years, according to market-research firm Almighty Institute of Music Retail. Last February, Tower Records, with ninety-three stores, declared bankruptcy and is now up for sale; Musicland has already changed owners, with many local outposts shuttered.

Wal-Mart is like no traditional record seller. Unlike a typical Tower store, which stocks 60,000 titles, an average Wal-Mart carries about 5,000 CDs. That leaves little room on the shelf for developing artists or independent labels. There's also scant space for catalog albums, which now represent about forty percent of all sales. At a Wal-Mart Supercenter in Thorton, Colorado, for example, there were no copies of the Rolling Stones' Exile on Main Street or Nirvana's Nevermind.

So there you have it. Scarcity, bottlenecks, the distortion of distribution and the tyranny of shelf space all wrapped up in one big store. Ironic that a place that seems to have so much could in fact have so little in each category. It's the paradox of plenty: a mile wide and an inch deep may look like everything at first glance, but in a world that's actually a mile wide and a mile deep a veneer of variety is not enough.

How to get filthy stinking rich in the Long Tail

Here's a surefire way to make a mint, or at least impress some investment bankers. One of the consulting request I get (which I never really satisfy) is to help companies or prospective investors reassess content archives in light of the Long Tail. In other words, if an archive was worth $X million yesterday, what will it be worth tomorrow, when there are so many more efficient ways to find and reach an audience?

My semi-serious suggestion is to use the following three-step process.

1) Plot the current demand for that content on a log-log scale, like this, which happens to show box-office revenues for 2004 US movie releases. (Thanks, Kevin Marks):

2) Assume that the natural shape of demand is a straight line, as any powerlaw will be when plotted log-log. Thus the fall-off seen in the real world data must reflect artificially suppressed demand due to distribution inefficiencies. (In this case, it just happens to be that the movies after the first 1,000 or so just didn't get much theatrical distribution.) Therefore, the latent demand in that market is the area between the ideal straight line and the empirical data in the current marketplace. Like this:

3) Estimate the dollar value of that red bit. That's the additional value of any archive in a Long Tail market. Bid quite a bit less than that over the current price for the archive. Sit back and consider how to spend your forthcoming riches.