The Long Tail

Could the labels actually be right?

Although it's tempting to assume that the evil record labels are once again trying to gouge us, there's some sense in their latest efforts to get Apple to abandon it's one-size-fits-all pricing model. A New York Times article over the weekend reported on the ongoing struggle between the labels and Apple over its fixed $0.99 price point. The labels would like to sell most new music for more--$1.49/track?-- while older or more obscure tracks could go for less.

There's plenty to like about variable pricing. For starters, it's almost always the most efficient way to maximize markets of disparate goods and customers. As Barry Ritholtz puts it:

It's a basic rule of economics: goods that have elastic demand  (i..e, non essential) are highly price sensitive. Further, any item easily available for free (albeit illegally) will have an even bigger response to price increases.

Apple has argued that single-price simplicity was necessary in the early days of the service, when people were just getting used to paying to download music. But now, after 500m tracks have been sold, we're clearly past the early adopter phase. So what's the right pricing model going forward?

Most accounts of the dispute between Apple and the labels have focused on the industry's efforts to raise prices, which are undeniably a big part of their plan. No surprise there. The research we've been doing for the book shows that within the bulk of the online music business--the top 100,000 downloads--only 3.5 tracks on the average CD sell. So the record labels are getting less than $3 in revenue (wholesale) from albums when the music is sold by the track. That's less than half the wholesale price of a CD (although with none of the physical costs of making and distributing a CD). The shift from an album model to a track model is indeed an alarming thing for the labels, and it's easy to see why they'd want to raise retail prices online as a result.

But there's more to the story that that. The labels may be evil, but they're not (all) stupid. They--to say nothing of many of their artists--also see the virtues of dropping the price for lots of their music, too. For decades they've been playing with CD pricing models that range from cut-price classics to top-dollar boxed sets, and when freed of the overheads of traditional retail, they're likely to experiment more, not less. Although some of the more vocal commentators have encouraged Apple to hold the line at $0.99, there's a strong argument that introducing variable pricing might ultimately lead to a more consumer-friendly outcome.

The reason is simple Long Tail math: there's a lot more music in the Tail than there is in the Head, and labels are generally more willing to experiment with discount pricing outside of the top 1,000 than they are with their hits. Those niches represents most of the music available today, measured by number of titles, and because they're only modest sellers individually they're less likely to create channel conflict with CD retailers, who tend to only stock the hits.

Imagine, for starters, that Apple introduces a three-tiered band of pricing: $1.49, $.99 and $.79 (that would no doubt soon expand to include $.49, but below that the transaction costs of credit card processing and the like start to loom large). Tiered pricing--gold, silver, bronze--is still pretty simple for consumers to understand, yet it introduces a valuable new dimension of demand creation.

Rhapsody, for instance, saw demand triple last year when it cut prices in half, to $0.49. And the average usage per customer in the all-you-can-eat subscription services is typically 5-10 times that of the pay-per-download music stores, such as iTunes. Add to that the potential of free music, which Apple has already paved the way for with its podcast service and could easily be extended to music licensed under Creative Commons, and you could grow grow the user base of the commercial music services manyfold.

Once iTunes allowed variable pricing, I imagine labels would do pretty much what they already do with CDs: set a range of wholesale prices depending on age, popularity and target market. Some labels will unwisely try to jack up all their prices and will eventually suffer lower overall revenues as a result. But smarter labels will experiment with all sorts of pricing variations, from lower prices for bands they're trying to break to volume discounts for packages of older music, and even higher prices for special reissues and repackaging for hard-core fans. That's already what they do in the CD world, where even standard retail prices range from around $9 to $18.

Apple may think it's protecting us from record label avarice with $0.99, which may be true in the short term. But long term, one-size-fits-all pricing is just constraining the economics of the industry and holding back the market. If Apple introduced variable pricing, it's not hard to see how the average price might actually fall in a year or two, thanks to the number of titles in the discounted niche/backcatalog categories vastly outnumbering the more expensive hits.

As long as prices can go down as well as up, I'm confident that market forces will eventually reveal the right set of models. And I'm even more sure they will confirm that no one model is right for everyone and every song.

"Just enough piracy"

It's not news that the main reason the movie and television industries are wary of BitTorrent is that they're freaked out by the music industry's experience with piracy. Although they see the economic advantages of P2P distribution, they're concerned that once they put their stuff out there, even wrapped in triple layers of kryptonite DRM, it might be cracked and then circulate in unprotected form. For movies, that's lost revenues. For TV shows, that means ads could be stripped out, expiration routines could be removed and  (gasp!) content could be modified or remixed.

All that counts as Very Scary Stuff to industry executives, and as a result they're looking for "strong" DRM before they consider letting their premier content circulate online. This is a mistake, for two reasons:

The first is about the user experience: Any protection technology that is really difficult to crack is probably too cumbersome to be accepted by consumers.

We've seen all sorts of failures of this sort before, from dongles to laborious and confusing registration schemes. Each seems better at annoying consumers than at building markets. The lesson from these examples is that zero-percent piracy is not only unattainable, it's economically suboptimal. If your content is uncrackable, it means you've probably locked the market down so tight that even honest consumers are being inconvenienced.

Instead, efficient software and entertainment markets should exhibit just enough piracy to suggest that the industry has got the balance of control about right: not too loose and not too tight. That number is not zero percent (which requires protection methods so invasive they kill demand), and it's not 100% (which kills the business). It's somewhere in-between.

The second reason the quest for zero-piracy is a mistake is an economic one: piracy can actually let you raise your prices.

I'll give you a surprising example. I was chatting with a former Microsoft manager the other day and he revealed that after much analysis Microsoft had realized that some piracy is not only inevitable, but could actually be economically optimal. The reason is counterintuitive, but intriguing.

The usual price-setting method is to look at the entire potential market, from the many at the economic lower end to the few at the top, and set a price somewhere in between the top and bottom that will maximize total revenues. But if you cede the bottom to piracy, you can set a price between the top and the middle. The result: higher revenues per copy, and potentially higher revenues overall.

(This is, by the way, the opposite of the conventional economic approach to developing-world piracy, which is to lower the cost of a product closer to the pirate version, closing the pricing gap to try to win customers over to the official version. In practice, however, the pirate price is so low that it's rarely possible to close that gap enough to make much of a difference.)

Add to this the familiar (if controversial) argument that piracy helps seed technology markets, and can be a net benefit. Especially in fast-developing countries such as China and India, the ubiquity of pirated Windows and Office have made them de-facto national standards. Few users could have paid for the retail versions at the start, but now that the spread of cheap technology, including free software, has led to an economic boom, Microsoft is finding a nice market for commercial software at the very top, in big companies and government offices.

When all these effects are considered, it appears that there actually is an optimal level of piracy. That right level would vary from industry to industry. Today the estimated piracy rates are 33% for CDs and 15% for DVDs. The industries say that's too high, but most anti-copying technologies they've brought in to lower that figure have proven unpopular. Would even tighter lock-downs help? Probably not. Maybe 15%-30% is simply the market saying that this is the optimal rate of piracy for those industries, and any effort to lower that significantly would either choke demand or push even more people to the dark side.

So the moral for video content holders and others considering DRM: be careful what you ask for, because you just might get it. "Uncrackable" DRM could make the P2P problem worse, by driving more users underground and depressing prices. Don't imagine that if you release content in a relatively weak DRM wrapper (like today's DVDs) and copies get out that the whole market will collapse. Instead, you may find that piracy stays constant at relatively low levels, leaving the rest of the market happier and more profitable.

The lesson is to find a good-enough approach to content protection that is easy, convenient and non-annoying to most people, and then accept that there will be some leakage. Most consumers see the value in paying for something of guaranteed quality and legality, as long as you don't treat them like potential criminals. And the minority of others, who are willing to take the risks and go to the trouble of finding the pirated versions? Well, they probably weren't your best market anyway.

Talking with Bram Cohen about P2P TV

At Foo Camp yesterday I was chatting with Bram Cohen (remixed version at right), the inventor of BitTorrent, about what it will take for the TV industry to embrace P2P distribution. Unlike music and movies, where the industry is terrified of piracy and lost revenues, TV could easily use P2P to extend its "free-to-air" ad-driven business model.

The logic goes like this:

1. Most TV is already free, so nothing is being "stolen" in file-trading.
2. Most television content owners want to reach more viewers for their advertisers.
3. One way to do that is via more distribution, which is expensive in broadcast
4. BitTorrent provides distribution for free

So if there were a way to let the content circulate on the P2P networks with advertising intact, this would seem like a Good Thing for all involved.

There are, however, another four reasons why this isn't quite as neat a fit for the networks as it appears at first glance:

1. Networks rightly assume that people will strip out the ads and circulate "clean" versions.
2. Even if the ads remain, most are designed for a specific time and place. "Now on sale" may not be true weeks later, and local ads don't make sense elsewhere. Unlike broadcast, P2P content can be viewed at any time (yes, the same is true of DVRs, and the advertisers aren't happy about that, either).
   
3. Local network affiliates often have geographic exclusives to certain shows. BitTorrent ignores geography.
4. Networks are concerned that widespread file trading of television content would compete with the fast-growing market for DVDs distribution of TV series. Although only a fraction of TV makes it to DVD, they don't always know which shows will make the cut at the start, in the broadcast window, so they don't want to limit their options by letting the files circulate online.

One can, however, imagine solutions to all of these:

1. Content owners could release the video in a protected form that makes it difficult to strip out the ads (you could even make it difficult to skip them). That packaging could also be smart enough to report back when played, giving networks more solid viewership numbers than they can get in broadcast. Alternatively, networks could just rely on revenues from product placement in the content, which is impossible to strip out.
2. They could only use ads that work well nationally and over a period of weeks. These tend to be ads for brands, rather than specific products, but there are plenty of those.
   
3. Quoth Bram: "Stop doing geographic exclusives." Indeed, many networks are moving away from that.
4. Make those protected videos expire after six months or so, so they won't be around to compete with the DVD if and when it comes out.

Let me be clear that these are my arguments, not Bram's. He's no fan of DRM (more on practical than moral grounds) and is hoping that more content-owners come around to the benefits of letting their video circulate freely without restrictions. And there some sign that's beginning to happen, with new examples including that of ADV Films, which distributes Anime in the US, using BitTorrent distribution for trailers and other promotional material. Bram now has a half-dozen people working with him to further commercialize the technology, making it easier for others to follow ADV's example.

I suspect it will take some form of content protection, such as the above, to get the networks to follow suit (they are, as you might imagine, incredibly risk-adverse when it comes to their golden geese). But one way or another, that day will come. The economic efficiencies of BitTorrent's distribution model are too compelling, especially for non-mainstream fare. The question is not if, but how and when P2P TV takes off.

Are niches more profitable than hits?

After I posted my new 80/20 graphic last week (thumbnailed at right; click to expand), a number of readers queried the bar on the bottom right, which shows that in Long Tail markets profits can follow revenues equally into the niches, something that isn't true for traditional retail and is thus a big deal. But readers think I've actually understated the effect and the advantages are even more dramatic than I showed. And it looks like they're right.

I've argued that profit parity is a key advantage of the Long Tail. In bricks-and-mortar stores, economics favor the hits because they use shelf space so efficiently, briskly whizzing off the rack nearly as soon as they're placed there. But in Long Tail markets, where the costs of shelf space are very low, the niches have the same costs as the hits, and potentially the same profit margins. This explains that last profit bar in this graphic, which is the same as the revenue bar that comes before it.

But I underestimated the effect of lower niche content acquisition costs. The readers were right. The economics of niches turn out to be even better than the hits. Way better, as it happens.

To see why, let's look at the DVD retail market. Here's a rough sense of the financial picture (the estimates are based on input from a number of retail experts, including William Fisher of DVDStation):

What you see here is that the economics of new releases these days are simply awful. The studios charge $17-$19 for the DVDs and the "big box" retailers (Wal-mart, Best Buy) sell them for $15-$17 for the first week or two, for an average loss of $2 per DVD (this is before overheads; the actual loss is larger).

After the first month or so, the wholesale price of the DVDs goes down faster than the retail price, and they gradually move into profitability. Yet 70% of DVD sales are of titles within their first two months of release, before they're profitable. Why do stores sell new releases so cheaply? Because for the big-box retailers, at least, they're a loss leader, designed to draw people to other titles in DVD section and elsewhere in the store, where the margins are better.

DVD distributors encourage this by allowing unsold new releases to be returned, lowering the risk for retailers (but increasing it for the studios, as Dreamworks and Pixar have just learned to their cost).

The problem is that while this makes sense of the big-box retailers who have other things to sell, it has the effect of setting the price for everyone else, including the specialty DVD retailers like Blockbuster. The big-box retailers have thus driven down the margins for new releases across the industry, making the economics of the Head even tougher. No wonder Blockbuster's stock is down 50% this year.

But if you could shift demand further into the Tail, creating a market that wasn't so dependent on new releases, you could improve the profit picture immensely. People move in herds, so this doesn't happen overnight, but it's not impossible. This is why recommendations and other filters are so important to Long Tail markets. By encouraging people to venture from the hits world (high acquisition costs) to the niche world (low acquisition costs), smart retailers have the potential to improve the economics of retail dramatically.

(This is, by the way, exactly what Netflix does: It underbuys new releases, despite the fact that such unavailability and delay annoys some customers and increases churn, because it allows Netflix to maintain its margins.)

Note that while I've given the case of DVDs, the exact same is true for music and books, and probably a lot of other things. The bulk of the revenues may still be in the hits, but increasingly the profits are in the niches.

So, chastened by this lesson, I've redrawn that graphic, reflecting the improved profit picture in the Tail (the numbers are for illustration purposes only; they would vary in diferent markets).

More evil "Headism"

It's amazing how often we fall into the trap of hit-centric thinking. I gave a few examples of this in a previous post on the Dangers of  "Headism" (or should that be "Hitism"? Advice in the comments, pls), but more examples pop up each day.

Today's example is the mistake of seeing the world through Top 10, 100, and 500 lists. There's nothing wrong with ranking by popularity--after all, that's just another example of a "wisdom of crowds" filter--but all too often these lists lump together all sorts of niches, genres, sub-genres and categories into one unholy mess.

Case in point: the Feedster Top 500 blogs. It gives the top ten blogs (by incoming links) as:

1. Engadget
2. deviantART.com
3. Boing Boing
4. Albino Blacksheep
5. Daily Kos
6. The News is NowPublic.com
7. Fleshbot
8. Gizmodo
9. Michelle Malkin
10. PostSecret

Meanwhile Technorati has updated its Top100 list (also ranked by incoming links) and the top ten there are:

1. Boing Boing: A Directory of Wonderful Things
2. Daily Kos: State of the Nation
3. Drew Curtis' FARK.com
4. Gizmodo: The Gadgets Weblog
5. Instapundit.com
6. Engadget
7. PostSecret
8. Talking Points Memo: by Joshua Micah Marshall
9. Davenetics Politics Media Musings
10. dooce

What have we learned (other than that the methodology of counting incoming links is still a messy science)? Well, not much. There are a couple of gadget blogs in each list, two or three political blogs, some uncategorizeable subculture ones (BoingBoing, FARK), a personal blog (dooce) and a Flash developers site (Albino Blacksheep).

These lists are, in other words, a semi-random collection of totally disparate things.

To use an analogy, top-blog lists are akin to saying that the bestsellers in the supermarket today were:

1. DairyFresh 2% Vitamin D Milk
2. Hayseed Farms mixed grain Bread
3. Bananas, assorted bunches
4. Crunchios cereal, large size
5. DietWhoopsy, 12-pack, cans
6. and so on...

Which is pointless. Nobody cares if bananas outsell soft drinks. What they care about is which soft drink outsells which other soft drink. Lists only make sense in context, comparing like with like within a category.

Jeff Jarvis made this point nicely yesterday:

When Ad Age gives you lists of magazine revenue, it separates women’s and entertainment and business publications; in big-media, those pass as niches and they are far more valuable comparisons. When talking about newspapers, you don’t lump in metro papers with town papers with trade papers; it’s a meaningless lump.When somebody can tell me who the queen of the knitting bloggers is, then I’ll listen…. and so will knitting advertisers.

And now ClickZ is complaining, as well:

[The Feedster list] packs dubious value as an evaluation tool for media buyers, according to several agency executives who spoke with ClickZ News. That's because it doesn't rank blogs according to niche or topical focus, wherein lies their main appeal to marketers.

My take: this is another reminder that you have to treat niches as niches. When you look at a wildly diverse three-dimensional marketplace through a one-dimensional lens, you get nonsense. It's a list, but it's a list without meaning. What matters in the rankings within a genre (or subgenre), not across genres.

I've addressed this a bit in past, arguing for "fine-sliced aggregators" and proposing a methodology to look at the Long Tail as a collection of "minitails", each best addressed at the niche level.

Which brings us back to the music service examples that we've been talking about here. The problem with one-size-fits-all services is that they treat all music the same way, which suffers from lowest-common-denominator problems. The genre-specific ones, meanwhile, are optimized for one kind of music but don't integrate easily with the big world of music outside that genre. The question, and the big opportunity, is how to do both, while avoiding the apples-and-oranges problems that blog lists and other hit-centric ranking exercises fall into.

How to make a better music service, part II

Thanks to all of you who have written with examples of sites and services that offer varying degrees of what I asked for in yesterday's post. It appears that although there are lots of bits of pieces of this out there, you can't (yet) have it all.

Today you can have: A) abundant information, rich metadata, and the ability to organize the library any way you want, but not in a full-featured music service with the industry-standard track downloading, streaming and custom radio channels. Or: B) all the standard commercial music service features.

But you can't have both.

In category A, here are some of the interesting music information services that, together, offer a good bit of what I asked for. But note that none of them are, at least in my quick review, full-featured listening/buying services:

Allmusic.com (great info, but no streaming, downloading or radio)
MusicBrainz (wiki-style, no music listening)
Last.FM (incorporates audiscrobbler; just radio)
upto11 (no music listening)
Discogs (no music listening)

There are also a bunch of music recommendation services, such as Pandora (formerly SavageBeast), MusicStrands, and LivePlasma.

Why isn't the great information in these services incorporated into iTunes, Rhapsody, Yahoo! Music, MSN Music, or Napster? After all, it works terrifically well for movies on IMDB, and that's part of Amazon. Why don't we have the same for music?

In an email (posted with permission) Barry Ritholtz suggests four explanations:

1. There is much, much more music than film, so [any rich database that attempts to covers most of it] may be a bit "kludgy";
 
2. Musical preferences are so much more personal and less communal experience than film; Most music recommendation engines have been less successful than that of movies; [Chris: I'm not sure I agree with Barry on this.]
 
3. The language of Film is far more accessible than that of Music in our modern era of Irony. Ever since Animal House and Caddyshack, we use film quotes as a shorthand for nearly any situation we encounter. Not so with Music (and its why a good musical director for a film can make such a difference);
 
4. The lingua franca of Music is so much broader and deeper than film that it encourages smaller niches. [Chris: I'm not sure I agree with Barry on this, either.]

In a comment to a previous post on this subject, Ryan Shaw suggests a darker explanation:

One reason I think they haven't seen wider exposure is that powerful forces like Gracenote are threatening companies who consider working with these open projects, seeing them as a threat to their desired metadata monopoly. Not to mentioned misguided record companies who consider any information related to their properties to be 100% owned by them.

And finally, several people have noted the rumor that Google may be announcing some kind of link-up with iTunes. I wouldn't expect too much of that in the near term, but it's clear that this is exactly the kind of problem that Google's great at solving. The information is out there, people want it, and it's simply a matter of connecting the supply with the demand to create a big new market. I hope it happens sooner rather than later.

A better way to find music

It's such a thrill to have one-click access to practically any song ever released that it seems churlish to ask for more. But I will anyway. Why are the current generation of music services so dumb?

    Recommendations, playlists and even detailed genre-level organization (which Rhapsody does best) are great, but they're not enough. I want to reorder the world of music my own way, and my way is different from the next guy's way. In the movie world this is easy, because we've got IMDB, which demonstrates what extensive cross-linking of every pertinent fact, from each cast and crew member to all the companies involved, can offer. So why don't we have the same for music?

    There's a world of cross-indexed value to be had in the connections between bands, performers and album. Think: "other bands of this genre that were playing in Washington DC from 1980-1985". But the iTunes, Napsters, Yahoo Musics and even Rhapsodys of the world offer almost none of it. Even in those rare instances when an album entry has more than the usual bare-bones metadata, it's not hyperlinked and thus you can't reorder the organization on the fly to find other relevant music. It is, for instance, a total mystery why you can't sort by record label on any of the main commercial services. Want to hear more from the Wax Trax stable? Good luck.

    Here are a few data fields (above and beyond artists and title) that I'd love to see clickable and attached to any track, allowing me to filter the rest of the collection by any combination of them to create different lenses on the music world that might make for a more satisfying music discovery process: All that information is out there somewhere; if the record labels can't supply the necessary data, then some smart music service should create a Wikipedia-like system of user-generated content to gather it themselves.

• Influences
• Home town (especially for local scenes)
• Active years
• Label
• Producer
• Spin-off bands (other bands with some of the same members).
• "Movement" (similar bands from the same time and place)

For classical, it would be a different list, including these:

• Composer
• Conductor
• Soloists
• Orchestra
• Label

Several commenters on my last post on this subject suggests some sites that are doing a better job at this than the big commercial services. Discogs, for instance, is trying something in this direction with a user-created discography, but it's still pretty ragged so far and doesn't offer a way to buy or listen to the music directly. Upto11 integrates some Wikipedia info, but likewise only offers buying links to Amazon or iTunes. I admire what these efforts are trying to do, but ultimately they've got to be integrated into a full-feature music service. It's 2005--why hasn't this been done already?

The 80/20 Rule Revisited

Several people have mentioned that our latest estimate of between 20% and 36% for Amazon's Long Tail book sales is "not far off from the traditional 80/20 rule," suggesting that this somehow minimizes the significance of Long Tail markets. I think this is a mistaken reading of the 80/20 rule in the Long Tail context (although that's easy enough to do, given that I've been guilty of it myself from time to time). Let me try to help clear it up here.

    The 80/20 rule is one of those phrases that means pretty much whatever you want it to mean. But in retail, it typically refers to the rule-of-thumb that 20% of products in a category account for 80% of the sales. It's basically the economics of hits: a small number of bestsellers account for most of the business. Such skewed distributions are common to almost all markets, even Long Tail markets (although they're not as skewed)--it's simply a function of powerlaw distributions, which are ubiquitous.

    What's different in Long Tail markets is the consequence of that skewed distribution. After failing a few times to describe clearly in words how the Long Tail changes the 80/20 rule assumptions, I've drawn this graphic in hopes that it will be clearer:

    What this graphic shows on the top is a simplified traditional retail scenario in which 20% of the products account for 80% of the revenues and virtually all of the profits (because high-turn products use shelf space more efficiently). But in Long Tail market at the bottom, where shelf space is infinite and the cost of carrying a niche product is roughly the same as carrying a hit product, three things change:

1. You can offer many more products.
2. Because it is so much easier to find these products (thanks to recommendations and other filters), sales are spread more evenly between hits and niches.
3. Because the economics of niches are roughly the same as hits, profit is spread as evenly as sales.

    Now let's apply this to the Amazon case. We've revised our estimate of Amazon's Long Tail sales, from 57% to 25%-36%. This refers to the portion represented by yellow bars in the graphic above--sales of books not available in bricks-and-mortar retailers, such as Barnes & Noble. (The graphic is just representative; it's not real data. But for the sake of argument, you can imagine that we're using the low end of our current estimate and the Long Tail book sales are the shown 25% of total sales.)

    What are the consequences of this change? For starters, it simply puts Amazon back in line with the other Long Tail retailers we've analyzed, Netflix and Rhapsody, both of which have between 20% and 30% of their sales in products not available in their main offline competitors. Note that in the year since we began our research on this, those numbers have all gone up. Rhapsody, for instance, went from 23% to 28% in that year. So although we corrected an analysis error in Amazon, the numbers are still significant and are getting larger.

    Secondly, don't think that sales of books not available offline is the only Long Tail effect. Amazon also sells more of the niche books that are available offline (thanks to its recommendations, search, reviews and other features that have the effect of flattening the demand curve), and it makes more money on those niche sales because they don't consume scarce shelf space. Combined, you get the profit picture in the bar on the bottom right, which is the result of all these effects.

    Now let's go back to the 80/20 rule.  One way of looking at it is to observe that the Long Tail effect actually makes the 80/20 phenomenon worse. Now, because you have so many products available, the hits that still represent most of the sales become an even smaller fraction of the total inventory. For instance, in the case of Rhapsody, which has a million tracks, it's closer to an 80/8 rule: just 8% of the songs account for 80% of the sales. That's what happens when you can add products without limit to your inventory; the percentage that sell a lot shrinks.

    Which leads to an important Long Tail lesson: it's not about about percentages. It's about absolute sales. Each niche item may not sell a lot, but because there are so many of them and because you can sell them so efficiently, the aggregate sales over all the niches can add up to a big new revenue--and profit--source.

    My original thesis regarding the 80/20 rules was the following, which is a bit more subtle than simply the end of 80/20 (despite my occasional bombast along those lines):

"The products that represent 80% of sales at a traditional bricks-and-mortar retailer will account for just 50% of sales at a Long Tail retailer. The rest--including all the products that are not available in traditional retail at all--will account for the other 50%."

  You can see this in the above graphic. The 20% of products (red area in the top left) represent the 80% of sales (top middle) in traditional markets, but those same products are just 50% of sales in Long Tail markets. I sometimes sloppily shorthand that by saying "the 80/20 rule is turning into the 50/50 rule". But since that shamelessly changes the definition of the terms from one end of that sentence to the other, I can see why there's been some confusion. My bad.

    You might wonder whether this thesis is turning out to be true as we collect hard data. So far, it's looking pretty close. The top 1,500 albums represent 80% of  CD sales in traditional retail, such as Wal-Mart, but only 40% of the sales on Rhapsody. Netflix data is looking closer to 50%. And, once we get our Amazon methodology more solid, I'll be able to test the thesis against that, too. I'll let you know.

Amazon methodology update

Erik Brynjolfsson and Michael Smith, the MIT economists who led the team that did the research we used as the basis of our original Amazon Long Tail sales estimate, have responded to my previous post on a new methodology for estimating Amazon's Long Tail sales. Bottom line, their latest thinking and work on this still suggests an estimate between 35% and 40%, which is at the higher end of Rosenthal's range. But if top-100 sales at Amazon are much higher than the model suggests, the number could be closer to the mid-20%s. The next step in this research is to gather more hard data, especially for bestsellers, to refine the models. Fortunately it looks like we may be able to get that in time for the book.

Professors Brynjolfsson and Smith write:

The blog is a great way to have a discussion on the growing importance of the Long Tail and it was nice to see Morris Rosenthal update the analysis. Here are some quick thoughts on the Amazon's long tail to add to the discussion:

1. In our original 2002 estimates in the paper with Jeffrey Hu, we used data from a large book publisher (who asked to remain anonymous) who provided detailed data correlating weekly sales quantities with average weekly sales rank for 321 book titles tracked over several weeks in the summer of 2001. We end up with a total of 861 points.  They include ranks lower than 250 and up to about 1,000,000. We found that these points fit remarkably well to a Pareto (a.k.a. log-log) curve and used that to come up with our estimate that 39.2% of Amazon's book sales fall in titles with ranks above 100,000. (Incidentally, 100,000 is the number of unique titles at a typical Barnes and Noble Superstore; the number is closer to 40,000 for the average bookstore.)

2. This estimate relies on an assumption that the Pareto curve does a good job of approximating the both the very top (ranks <250) and very bottom (ranks >1,000,000) of the curve. If this isn't true it may over or underestimate true sales. It also relies on an assumption of the number of unique titles at Amazon, which we took to be 2,300,000 in our paper (which at the time was the number of books in print). Using 2,000,000 would give an estimate of 37.8%, using 2,500,000 would give an estimate of 39.9%.

3. Others have created similar estimates. Most notably, at about the same time as our work, Judy Chevalier and Austan Goolsbee developed an experiment to approximate the fit of a Pareto curve by purchasing titles from Amazon and observing the sales and ranks before and after the experiment. They also cite two other independent estimates available at the time using similar experiments. (You are too generous in giving us the credit for developing this technique in your original blog entry, although we did incorporate a version of it into subsequent drafts of our paper.) We think the larger sample of data from the publisher is more reliable, although both approaches yield broadly similar estimates, as we note in the paper. In mid-2004, Michael Smith, conducted his own calibration of Amazon's Pareto using the experiment-based technique. Applying the Pareto estimates from these four sources to our calculations above would give Long Tail estimates ranging from 27.7% to 44.5% of Amazon's sales.

5. Thus, we believe estimates in the range of 35-40% are probably the best place to start. This encompasses both our initial estimate of 39.2% and Mr.Rosenthal's estimate of 36% using Amazon's most recent ranking algorithm.

But you could convince us of estimates as low as 25% if the Pareto assumptions are violated in the highest or lowest ranked books. (However, we found our original data matched the Pareto assumptions well, and if anything had a slightly fatter than normal long tail.)

In any event, I agree with the spirit of your blog comments -- let's not lose the forest for the trees. Wherever the true number lies in this range, the value generated by Long Tail markets to consumers and smart producers is substantial. In our original work, we placed the value of access to "The Long Tail" in books at around $1 billion per year to consumers.  It's undoubtedly much higher now. And this value -- combined with value generated from the other long tail industries that your work has identified -- is the real story.

We're continuing to do some work in this area with new data and methods.  We don't have anything to report just yet, but early results suggest that we're all on the right track regarding the growing importance of this phenomenon.

Mark Cuban's blind spot

[Updated; see below.]

I know I'm on record as being something of a Mark Cuban fan, but this time he's just got it wrong. He picks a fight with George "The End of Television" Gilder, which isn't hard. But although he starts off well, citing Aaron Spelling's aphorism that “TV is the path of least resistance from complete boredom”, the logic quickly goes off the rails:

George and others seem to think that unlimited choice is the holy grail of TV. It's not.

The reason it's not is that it's too much work to page through an unlimited number of options. It's too much work to have to think of what it is we might like to watch. We are afraid we might miss something that we really did want to watch. Put another way, it's way too hard to shop for shows in a store where the aisles are endless. It's stressful and a lot of work. Which is exactly why when we channel surf, or when we surf the net, we all end up surfing the same 10, 15 , 20 channels/sites over and over again. It's the path of least resistance.

    Right there you see the problem. This is what people said about the Internet--"there's too much there; it's too hard to find what I want"--back in the days before Google. Today we don't "page through an unlimited number of options" anymore. We either search and let the software rank results in order or relevance, usually showing us what we want in the first page, or we let recommendations suggest stuff we'd like, as in the case of iTunes and Amazon. Indeed, Cuban mentions how much he loves shopping at iTunes and Amazon for just this reason. Surely successful Internet TV will work the same way?

    Well, yes, says Cuban:

When we get to a point that there are thousands of on-demand TV choices, we won't approach TV programming guides like we do a search engine, looking for a specific target.  That's too much work.  The  smart on-demand providers will present their programming guides more like Amazon.com or Netflix.com. Both of which do a great job of “suggestive programming”.

We will get a personalized page with options that it thinks we might like based on our previous viewing decisions. Then different categories of shows, within each we will see best rated, most viewed and newest added, along with “play lists” suggested by branded guides who make recommendations. All of these simple options will make it easy for us to make a choice with some level of confidence.  We won't feel like we are missing something and we will know that if we don't like the show, we can quickly go back to a point that makes it easy to find another selection.

    Sounds good. So what's the problem?  Recall that Gilder is not predicting the end of video programming, he's predicting the end of TV as we know it, which is to say broadcast TV. That's a finite number of channels trying to find content that can aggregate audiences big enough to make shotgun economics work.

    Internet TV, on the other hand, is just like iTunes and Netflix--an infinite number of individual content elements that are custom packaged or presented based on our interests, whether explicitly expressed through search or implicitly through our viewing habits or personal profile. It's the death of channels, not the death of video programming. Gilder happens to define "TV" as traditional broadcast TV (interrupted by ads), and Internet TV as something else. Far from dismissing it, he thinks that something else has a glorious future, pretty much along the lines that Cuban himself describes once he gets past his inexplicable scorn for infinite choice.

    In short, Cuban's right that in a world of massive TV variety, we'll need smart, personalized recommendations and "suggestive programming" to make entertainment easy. But he's wrong to suggest that's in any way incompatible with Gilder's model of infinite choice.

    What Gilder is criticizing is the lowest-common-denominator model required by the economics of broadcast TV, where a limited number of  channels have to aggregate the biggest audiences they can. Cuban seems to think that Gilder's alternative model is Google, where you have to know what you want and then search for it. But that isn't what Gilder's saying at all. He's agnostic about how people find what they like in a massive market of abundant choice. It could be smart program guides or personalized filters. His point is that, regardless of how people find it, they'll be more satisfied with what they eventually get in this world of plenty than they are now in the pre-filtered world of scarcity.

    I think Cuban and Gilder actually agree. Infinite choice doesn't have to mean the tyranny of choice. More really can be better. And one way or another, the traditional broadcast model is going to be replaced.

UPDATE:

After commenting here in defense of broadcast distribution, Mark has posted on his own blog a longer explanation of why he think broadcast will be with us for good.