I was discussing on a private mailing list whether or not this number is
for real. Someone complained it wasn't. Here's my response:

-------------

Your utter certainty regarding the flawed methodology of their results
is somewhat ironic given that you've done no apparent research into how
they came to those results, nor any suggestion of what a more accurate
number might be.

A real critique would go to the source of their data and show why it's
bad. I'll help you with that. I found this year's report by simply
updating the year in the link from last year:

http://www.ifpi.org/content/section_resources/dmr2010.html

The actual quote from there is:

> Estimates on the impact of internet piracy vary but are consistently huge in scale. IFPI, collating separate studies in 16 countries over a four-year period, estimated unauthorised file-sharing at over 40 billion files in 2008. This means that globally around 95 per cent of music tracks are downloaded without payment to the artist or the music company that produced them.

When read in the actual context, they seem to admit it's a hard thing,
and acknowledge they're estimating. Furthermore, contrary to your claim
that it was pulled out of thin air, they kindly cite a variety of other
sources. I haven't dug into them all, but one is:

http://www.ipoque.com/resources/internet-studies/internet-study-2008_2009

The paper is 14 pages long with lots of interesting data, as well as a
completely description of its methodology. A weakness of their data is
they didn't cover North America or Western Europe as a whole (maybe due
to data privacy laws?). But focusing on Germany alone (which seems
middle of the road in terms of its data compared to other regions), it
says 53% of traffic was P2P downloads (separately from VoIP), to 26% web.

So right off the bat, P2P downloads account for double the traffic of
*all of HTTP* across their not insignificant sample set.

It doesn't seem unreasonable to assume that, say, 90% of that P2P
traffic was pirate. (It's what it was designed to do, after all.) So
let's say 47% of all traffic was pirate content downloads.

As for legitimate content downloads, we could probably back into that by
estimating bandwidth consumed by iTunes using public sales numbers and
average file sizes. But let's say iTunes accounts for 10% of *all HTTP*
(which seems astronomically high). That would mean 2.6% of the internet
is legitimate content downloads (separate from streaming).

47 + 2.6 = 49.6% of the internet devoted to content downloads
47/49.6 = 94.75% content downloads pirated.

Huh, I didn't even plan that out.

Anyway, there are obviously problems with all that data, how it's
collected, how it's analyzed, etc. But to categorically assert that the
data and results are flawed and everyone involved in the process is
either actively lying or allowing themselves to be mislead -- without
providing any evidence or analysis to the contrary... let's just say
your own methodology could use a bit more scientific rigor.

-david

Really interesting NY Times chart here showing the relative value of all units sold across different media (CD, vinyl, digital, etc) since 1978.

Does anyone know a similar breakdown by number of units sold (independent of value)?  It'd be interesting to see a breakdown of units (or even by tracks) to compare the total volume of legitimate songs being purchased over time.  I wonder if it's constant, or gradually increasing, or quickly increasing with the advent of the internet?

For example, if we could see that the number of songs both being produced and consumed is steadily increasing -- while simultaneously revenue is decreasing -- it seems hard to argue that revenue is correlated with increased song production/consumption.

(At best it could be argued that revenue leads to higher quality songs, but that's a tricky one given that "quality" is entirely subjective.)

And if revenue bears no correlation with song production/consumption, then policies that protect and maximize revenue (eg, Copyright) become rather difficult to defend given the stated purpose of music Copyright to -- essentially -- increase song production/consumption.

-david

Having the increasingly tired conversation about whether the music industry stands a chance (it doesn't), and wrote up this response for why the problems that do exist in pirate tools are pretty insignificant compared to the alternatives:

"All those problems [incomplete catalogue, fake files, bad tagging, variable sound quality, porn ads, ISP throttling, the risk of being sued, guilt, etc.] are real, no doubt.  But let's not forget: despite those problems, pirates still own 95% of the music download market.  Don't you think if pirates really cared to deal with any of those problems (better than they already do), they could?

Indeed, don't you think those problems are pretty trivial compared to the vastly more substantial problems (that you fail to list) of dealing with the cartels?  Namely: crippling fees for companies, unreasonably high fees for users, and the occasional licensing turf war that wipes out all users outside of the United States?

Thanks to technology: music is now bits, and bits are now infinite.  These are revolutionary facts.  Why is it so hard to accept that this revolution is like any other: those in power suffer while a new power emerges.  This is such obvious stuff, I don't know how we're still debating it so many years *after* the revolution ended.

I mean, for a moment imagine the pirates are the good guys in this battle.  Imagine you're one of them, and you've already secured 95% of your terrain (eg, the global music download market), and are only dealing with the occasional isolated incident from rogue terrorist outfits (eg, RIAA vs Jamie Thomas-Rasset) or settling regional disputes (three strikes laws, lawsuits against TPB).  Sure the pirates could improve their interfaces.  And surely they are improving.  But where's the rush?  Wouldn't you think the war had been won long ago?

If anything, I bet they're more interested preparing an offensive push into new terrain: the global music streaming market.  And if stupid things like Pandora needing to shut down its international userbase -- creating a global demand for something that there is no legitimate way to buy -- then they'll have no harder time winning and holding that terrain than they have music downloads.

-david"

Somebody came up with a brilliant idea: let's make a new audio file format! After all, people have tons of complaints about MP3s, right? Like... oh, wait, actually there are very few complaints. Undaunted, and with a $2.5M war chest ($2.5M to create a file format!?), MXP4's advanced technology is poised to "revolutionize the music experience"... uh, what? That full quote:

So what makes MXP4 so advanced? The file format, beta-released in September, contains multiple tracks, allows users to mix the music, and incorporates video. On the mixing side, different track elements can be suppressed and recombined, allowing remixes, karaoke versions, or others creative combinations. "This are clear signs that the music industry is beginning to see the potential for MXP4 to revolutionize the music experience for consumers by allowing them to play with the music, whilst opening up new promotional and revenue possibilities for artists and labels alike," Serviant commented.

Riiight... I think this'll fail. Not (just) because it brings insignificant value. But for a reason that sounds incredibly trivial but is actually really significant: MXP4 has too many letters.

All successful file formats have TLAs. It's just how things work. Yes, technically you can have a four-letter file extension. Just nobody ever does it, so it looks really weird.

Even "jpeg" eventually dropped the "e" to become "jpg" -- and "jpeg" sounded fine when you said it out loud. MXP4 sounds incredibly awkward spelled out, and doesn't sound like anything when pronounced like a word. That means every website, tool, story, and mention of this abysmal product will be tainted with an awkward, unpronounceable tinge.

I think had they called it MP5, or even just MPX, they'd be in a far better position. But MXP4 is this weird bastard name -- it's not the clear successor to MP3 that MP4 connotes, nor is it even in the MP family (it's in some new MXP family). But rather than being the first of a new family (MXP1), it's spontaneously the fourth generation -- in an obvious ploy to sound better than MP3.

It's a name only a high-paid marketing team could come up with.

-David Barrett
Follow me at http://twitter.com/quinthar

(This is in response to an email from "Anton" suggesting that we're at the dawn of a new type of "dynamic recording.")

I actually really agree with you, if not on the specifics, but on the potential for a genuinely new type of music originating on the internet that is structurally unlike anything before -- and that is intrinsically incompatible with and stifled by copyright.

For example, remember that even the concept of "notes" was once an innovation.  Prior to that, music was a collection of sounds at various frequencies, without an awareness that certain frequencies just sound "better" (nor a mathematical understanding of why that's the case).  When "notes" were invented/discovered -- along with the technology to produce them reliably -- music itself fundamentally changed.

Similarly, some might have thought notes were the end of the line, but then came along chords.  Again, it was a real discovery that could only be enabled through technology: you simply can't do chords until you have the technical ability to generate multiple "notes" reliably and simultaneously.

(And if you haven't yet invented notes, then chords are simply impossible.)

Then the pianoforte comes along -- again, a technical innovation -- that opens up an entirely new type of music that simply couldn't be done prior.  I'm sure we could come up with a thousand examples (including the use of distortion as an instrument, which gave rise to heavy metal) of how technology not merely extended music, but genuinely changed it.

I think computers and the internet present another innovation in that sense.  Prior to the digital age, it simply wasn't possible to -- for example -- mash up hundreds of videos or thousands of songs to make a new song.  But that's now possible, and its core "building block" isn't frequencies, notes, chords, or even instruments.  Its building block is whole songs/videos.  It's an entirely new building block that couldn't technically be considered before.  It's an entirely new type of music -- sampling taken to the extreme -- enabled through an entirely new technology.

And next?  As Anton suggests, prior to the internet, making globally interactive music -- whatever that might mean -- simply wasn't an option.  We can't even imagine what the consequence of that will be, nor what new type of music that might enable after.

But what I *can* imagine is all that might be fundamentally incompatible with today's notion of copyright.  Indeed, we might look back on the attempt to copyright individual songs as silly as trying to copyright individual notes or chords.

Indeed, maybe the reason all music seems to sound the same today is because we're discovering there are certain classes of songs that actually *are* the same, and sound better, for reasons we don't quite understand now but someday will.  This might be the same process early musicians grappled with when first discovering the core notes and chords that we now view as so fundamental to music.

Maybe far from witnessing the death of music as the industry would have us believe, we're seeing the birth of a whole new generation?

After all, those prior building blocks were perceived as innovative, radical, or even threatening back in their day.  Why should our day be any different?

- David Barrett
Follow me at http://twitter.com/quinthar

As I wrote about previously, Expensify is doing (what I believe to be) some pretty innovative Twitter marketing.  However, from the very start we realized there's a delicate line between marketing and spam, so we set out some early rules to ensure we're on the right side of the line:

1) Keep it personal.  Only send messages from real people, to real people.  Leave the faceless boxes on Google and maintain the social foundation of Twitter.

2) Keep it timely.  A huge benefit of Twitter is you can go straight the people who are experiencing the problem at that exact moment.  Leave the huge backlog of past posters alone and stay focused on the present.

3) Keep it relevant.  The temptation is overwhelming to just blast this out to everybody.  But resist that temptation and focus on the people who are actually calling out for your thing.

That said, we were afraid then that others would cross the line, and it appears that's happening with increasing frequency.  Alas.

Unfortunately, I'm not sure what Twitter could do to thwart it.  Perhaps the easiest way would be to just add a "Spam" button to the Twitter interface and then kick off users who get too many relative to their post volume.  In Expensify's case, we get 4x more compliments than complaints (the above rules appear to work!), so I think we'd do just fine under such a scheme.

But it's still too early to predict how the Twittersphere will react.  What do you think?

- David Barrett
Follow me at http://twitter.com/quinthar

Here's a big data dump of stats (followed by analysis), for those who care about this sort of thing, from a March 2009 ars technica article:

- 17M people stopped buying CDs in 2008

- 8M people started buying digital music in 2008
- There are now 36M digital music customers
- 1.5B songs were sold "digitally" (ie, online) in 2008  
- 33% "of all music tracks" purchased in the US were digital

- Pandora use doubled in 2008, to "18 percent of Internet users"
- "Social network music streaming" rose from 15 to 19 percent usage

A January 2009 ars technica article rounds out these stats with:

- "unit purchases" increased by 10.5% in 2008
- 428M albums (LPs + CDs + online) were sold in 2008, down 14%
- 65.6M online albums sold in 2008, up 32% over 2007
- 1.5B songs sold online in 2008, up 27% over 2007
- 1.88M vinyl sales in 2008, up 89% over 2007

So all that looks pretty rosy for the music industry, in absolute terms.  But how did it do relative to piracy?  According to this slightly more pessimistic January 2009 IFPI report:

- Digital music sales grew 25% in 2008 to $3.7B worldwide
- Digital music sales account for 20% of recorded music sales, up 15% over 2007

- 40B songs were "illegally file-shared" in 2008
- 72% of UK music consumers wold stop pirating if told to do so by their ISP
- 74% of French consumers agree internet disconnection is preferable to fines

A linked "key facts" PDF has a boatload of additional statistics, including:

- 16% of European internet users "regularly swapped infringing music" in 2008
- 13.7M films were distributed via P2P in France in May 2008, compared to 12.2M cinema tickets
- "free music" was given as the primary reason for piacy
- P2P file sharing accounts for up to 80% of traffic on ISP networks

So pirated downloads still utterly dominates legit downloads, to the tune of 26:1.  If anything, it seems like piracy is accelerating, even faster than legal download services.

What about legit streaming?  In July 2008 I estimated that MySpace users legally streamed about 110M songs per day.  Turns out I was off by a lot: they streamed 1B downloads after "only a few days", and this September 2008 TechCrunch article tosses out 20B streams initiated *per day*.  That's an amazing number.

But it's also an incredibly vague number, as stream initiation isn't nearly as interesting as stream completion.  For example, the average user spends under 10 minutes on the site per visit, meaning there's barely time for two full-length songs.  I'm having a surprisingly hard time finding recent data, but this 2007 article shows MySpace had like 29M daily visitors, so even doubling that for 60M daily visitors today suggests at most time for 120M full-length songs per day -- roughly 43B per year -- and this ignores the large subset of international users (who can't get newly-released music).

Similarly, YouTube had 5B views in July 2008, and 6B views in December 2008, so let's just assume something like 66B total videos in 2008.  As for what fraction of those equate to "songs" I have no idea; I'd say this is more about "intent" than anything (ie, people who play the video in the background like a radio, rather than watching it like a music video), and I have no data at all on that.  But I wager it's not the common case, so let's say 25% of YouTube videos are actually just played as songs -- and even that seems high.  (Also, this assumes all YouTube music is licensed, when in fact the opposite is probably more often true.  Details, details...)

Adding to MySpace's 43B and YouTube's 16.5B would be all of Pandora's streams, which should be considerable given the claim that 18% of all Internet users use it, but I can't find any data on it.  One reason for that is probably because Pandora actually has nowhere near that userbase: this Dec 19, 2008 TechCrunch article reports they only just hit 20M users, while in that same month the internet was estimated to comprise 248M North-American users (1.4B global).  This puts Pandora's penetration at a much more conservative 8% of North-American users (assuming 100% are North American), or 1% global.  Still significant, but 20M *total* users is nowhere near MySpace's 100M *active* users.

So for the sake of argument, let's say there are about 60B legit streams, against 40B pirated downloads -- meaning piracy utterly dominates in the download market, whereas legit streaming utterly dominates in the streaming market.  Indeed, there is essentially no such thing as a meaningful "legitimate" download market, or a meaningful "pirate" streaming market.

As for which accounts for more total "listens" and thus ultimately controls more users' ears, that's an open question: on the one hand, streamed songs are only heard at most once, whereas downloaded songs can be listened to multiple times.  But streamed songs are probably more likely to be heard at all, with a lot of pirated songs probably just going into vast personal libraries having never been played.

Who's winning?  Who knows, and as piracy goes dark, it's harder and harder to tell.  Personally, I'd still put my money on piracy having a strong lead on users' ears, both right now and for the forseeable future.  If the average pirated song is listened to just 1.5 times (which seems reasonable), than piracy is still winning.


So in conclusion, it seems to me that the battle for downloads is utterly and irretrievably lost to piracy, but the battle for pirate streaming is only just beginning.

As it stands, streaming is overwhelmingly in favor of legitimate content owners.  But I really wonder how long that will last. 

After all, the list of streaming P2P applications is long and always growing (now over encrypted onionskin darknets).  Basically, P2P streaming is a hard problem, but it's also largely a solved problem.  So if there's no technical reason why pirates don't stream, maybe they don't simply because they don't want to? 

The most obvious reason why this might be true is because people turn to piracy primarily to avoid paying.  (Please excuse the alliteration.)  So long as MySpace and YouTube continue give it out for free, there's little incentive to build a pirate streaming site.  But the real test will come if something in that calculation changes, by one or more of the major parties.

For example, let's say MySpace decides they don't like paying to stream content from central servers, and then paying again for licensing fees.  Maybe they find their ad revenue sagging and decide to integrate a streaming P2P plugin (I'm betting on Littleshoot for now) to offer the same exact experience as today -- but by tapping into the pirate networks.  So no bandwidth costs, no licensing fees.

Alternatively, let's say the powers that be do something incredibly stupid like pulling their music from MySpace, or jacking up the price such that MySpace is forced to charge for it.  At this point there's an opening for someone like The Pirate Bay to offer a first-class pirate station, and then it's game on.

Either party would use an argument like "we don't host any data, we just enable user sharing.  Any illegal behavior they do is their business and we don't encourage it (we merely profit from it)." 

And unlike the small P2P outfits who have tried this in the past, the next wave of defendants will have substantial legal resources and astonishing revenue incentive.  And unlike the tiny, outgunned P2P outfits of yore, MySpace's or The Pirate Bay's victory won't be quite so Pyrrhic.


Anyway, just wanted to do a quick review of the available data and update my predictions.  Can anyone provide more recent or accurate data to correct the above analysis, or see holes in the logic?  I'm as eager as anyone to get a firm grasp on reality; let me know if you think my grip is slipping.

Fun times, I can't wait to see where this goes.  Thankfully, it's going there really fast, so there's little time to wait.

- David Barrett
Follow me at http://twitter.com/quinthar

Every once in a while someone gets a brilliant idea for dealing with piracy: why not just assemble a big pool of money and then distribute it in proportion to how often content is pirated?

Both parts of that (filling the pool, and then selectively emptying it) are atrociously bad ideas for a huge number of reasons, but let me zero in on the latter half here.  In essence:

Under no circumstance proposed or envisioned will backbone measurement ever estimate volume to even the barest degree of accuracy, darknet or otherwise.

Consider what is ostensibly the most widely viewed image on the internet: the Google logo:

It's unprotected, unencrypted, no darknet, no P2P file sharing, no copying to an iPod for offline consumption.  In short, if backbone measurement could ever estimate *anything* then surely this would be the ideal use case, right?

But the Google image is cached locally -- in my case (according to about:cache in Firefox) until 2038.  No matter how many times I visit Google.com, I won't redownload it.  So estimating visits to Google.com by sampling the number of times the logo is downloaded is completely and irreparably flawed.

(And the most common caching solution is LRU so content that is accessed *more* often is actually re-downloaded *less*.)

Thus estimating the number of times a song is listened to by measuring how often it is downloaded is even more flawed -- as all the reasons I gave for why Google is the ideal case are precisely inverted for music.

Even if we can't agree on anything else, we should all at least agree that backbone sampling is a patently absurd notion for estimating popularity, and thus is intrinsically unsuitable for redistributing some big pool of money -- regardless of how it's filled.

- David Barrett
Twitter: Follow @quinthar

As predicted by many (including me), there's a new P2P network on the block with built-in onionskin routing: OneSwarm.

Even better, it's backwards compatible with BitTorrent, and they tossed in always-on "web-of-trust" encryption just for fun.

In English: what little light we ever had into pirate activity just got dimmer. And if we push them really hard, they'll go entirely dark.

If you thought 20:1 was hard to prove (or disprove) today, just *wait* until everything is encrypted and decentralized.

Next step: widespread adoption of decentralized tracking, followed by decentralized indexing -- perhaps using my good friend Tom Jacob's brilliant Localhost.

Keep pushing, RIAA. You're giving birth to a very angry child. And if you think it's painful now, just wait until it grows up.


-David Barrett

It's stories like this that convince me we're on to something so much bigger than copyright.  The cost of producing extremely high-quality content has come down so low, I think we're on the cusp of a much more exciting world than Copyright could have ever dreamed.

-david

Interesting article on TorrentFreak:

http://torrentfreak.com/p2p-researchers-fear-bittorrent-meltdown-090212/

Basically predicting a widespread meltdown if ThePirateBay goes under,
because they track over 50% of torrents. If TPB's trackers go down,
users will "fail over" to a bunch of other trackers that probably can't
handle the load, which will likely trigger a cascading failure of pretty
much all the trackers out there.

But what it doesn't mention is that this will probably all be fixed by
the end of the week and it'll be back to business as normal before the
end of the month.

In other words, the ultimate culmination of a multi-year, international
legal process against TPB will probably result in... a week or two of
disrupted downloading.

On top of that, if trackers start to get taken down with any regularity,
the various torrent client authors will probably just take the time to
perfect their "trackerless torrent" technology (generally based on
DHTs), and then they'll be even more indestructible.

And if everyone's going to upgrade, I bet they'll slip in "always on"
encryption (there goes any chance of backbone sampling!), and maybe some
early experimentation with onionskin routing.

Piracy will never be killed, and fighting it only makes it stronger.
It's like a self-fulfilling, cyclical prophecy -- the only consequence
of passing bills in the (disingenuous) name of "fighting terrorism and
preventing child pornography" is to encourage the creation of tools that
enable more of it, at no reduction to piracy whatsoever. Which in turn
fuels calls for more disingenuous bills, fueling more technology
development, and so on.

Call me crazy, but I am far more concerned that these P2P tools are
creating an untraceable infrastructure for *real* crime than for
pseudo-crime. One of these days there's going to be a huge story about
Iran coordinating with Hezbollah using encrypted P2P VoIP routed through
a decentralized onionskin network, or Al Qaeda distributing terrorist
materials using BitTorrent 3.0 -- and how the worlds' nations are
fundamentally unable to stop it... unless you give up more of your
rights to privacy, free speech, and other crucial civil liberties.

The RIAA has done more to pave the way for future terrorist
infrastructure than Bin Laden could ever dream.

-david

Fantastic graph in there showing the breakdown by music, movies, etc.

- 28.7% movies
- 22.8% music (I'm guessing mostly whole albums and compilations)
- 16.9% tv-shows
- 8.1% games
- 7.3% software
- 5.8% anime (why is this separate than pictures and movies?)
- 5.6% books
- 4.0% other
- 0.9% pictures

Interestingly: "the Pirate Bay has 765,000 torrents on their site, although they track twice as many files."

So TPB is skewed toward the head, which I guess makes sense.

-david

PS: IsoHunt and TorrentZ are listed, but their numbers aren't quite as meaningful as they are indexes, not trackers.  Many of their torrents are actually hosted on TPB and MiniNova.

Having (yet another) discussion on Copyright, and I'm starting to wonder about its core value in a world of excess.  The Constitution states the goal of Copyright (and its siblings) as:

To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.

So it's pretty clear that the whole value is promoting more of a good thing, which is only sensible based on a backdrop of not having enough of that good thing without it.  But I'm wondering, do we really not have enough?

Here are a couple musings to a private list on that topic.  First, regarding whether the notion of abolition is too extreme to even consider, versus just fixing what's there:

Totally agree about fixing before abolishing, in general.  But under what circumstances would you feel abolition is actually warranted?

We've already agreed that:

1) It doesn't accomplish its Constitutional objectives
2) It accomplishes unintended, damaging objectives

Were those alone, I agree, fixing might be in order.  But I'd also add:

3) Its original objectives have already been accomplished

This is not to say that all "science and useful arts" are done.  Rather, it's to say that they have been successfully promoted.  We're deluged with it.

Our patent office, once at risk of shutting down from a lack of applications, is overwhelmed with a multi-year backlog.  Our libraries and bookstores and iPods are full.

Nearly every US Citizen has nearly limitless access at nearly zero cost to nearly all human knowledge, 24/7, from their phone, nearly anywhere in the world.

What a luxury!  Our debate shouldn't be about encouraging the creation of more information -- you could read new books while simultaneously listening to new music and watching new movies, continuously, from birth to death, and never exhaust our supply.

Rather, we should be debating how to further expand the reach of existing information to every world citizen, in every socio-economic group.  And not just because it's so obviously the "right" thing to do, but because it will also make us even richer!

Basically, if today's reality doesn't call for the abolition of copyright, what reality *would*?  Or are you so wedded to the notion of Copyright that there is no universe in which you feel it shouldn't exist?  Do you feel it's an inviolate, human right -- so strong a right that all other rights should be discarded in its pursuit?

-david

PS: And before I'm accused of being a socialist, let's remember the alternative: creating a global tax to centrally fund the production of all art.  Similarly, it's not a call for communism: the most successful capitalist enterprises that deal with information deal with maximizing free access to unlimited information.

Then, in response to a concern that (roughly) "it's working so well and people are lapping it up, why stop now?"

The success of past promotion doesn't imply the need for more.

Nor does acceptance of government promotion imply its necessity.

If the government secured exclusive rights to -- say -- donut flavors, I would fully expect donut makers to accept it.  It's also possible such rights would contribute to a rise in the creation and consumption of donuts in our society. **

But that doesn't mean we need more donuts in the first place.

The question is whether Copyright is a means to an end, or a universal, human right.  If it's the former, then it's important to understand when that end has been reached, and to stop.  But we discuss it generally as if it's the latter -- as if no amount of promotion is enough or, heaven forbid, too much, no matter what the cost.

-david

** Though I wager it's more likely Dunkin Donuts and Krispy Kreme would just put all other donut vendors out of business.

Basically, when is enough enough?  Even if it was promoting useful science and art (probable), and if if it still is (incredibly unlikely), does that goal even matter anymore?

I know there's a dearth of real-world data on piracy, but here's a dash: the Pirate Bay recently passed the 25M mark in terms of simultaneous peers.  Here's the data itself, from TorrentFreak:

    Date            Peers (M)
    11/01/2006      1
    11/01/2007      6
    09/21/2008      15
    11/01/2008      20
    11/15/2008      25

That's interesting, but it looks much more compelling graphically:



So the rate of real pirates seen by The Pirate Bay has increased by 25x in just over 2 years.  (As for how meaningful this data is, it's suggested that TPB tracks about 50% of all torrents.)

Anybody care to guess whether that 20:1 ratio of illegal to legal downloads ratio is going?  Might it be more like 100:1 today?  The growth of piracy over legal downloads is just staggering.

-david

So I finally made it through to the Tribler website and I think it's *almost* really cool.  Don't get me wrong, I installed the app and it seems to do interesting stuff.  But I think they're biting off more than they can chew.  Basically, I'd layer the content-acquisition experience as follows:

Social: The experience of sharing and learning about new content
Index: Determining who has a given piece of content
Transport: Getting content from somebody else

Tribler tries to vertically integrate the entire stack, and they do a decent job, but it's just too much for one application.  It's like coming out with Prodigy today and trying to complete with "The Internet".  If Tribler came out before ThePirateBay then it'd be something.  But then it'd be called Kazaa.

No, instead I think they should do something less.  Indeed, the whole brilliance of BitTorrent was that it *didn't* do it all.  BitTorrent does nothing but standardize transport, which enabled a huge diversity on the upper layers.  Doing *less* than Kazaa is what made BitTorrent succeed.

So if BitTorrent is on one side, and Kazaa on the other, what should Tribler do?  I'd say stick with the old saying "only innovate one thing at a time" and just integrate the index into the transport layer, and then call it day.  Leave ThePirateBay and everyone else to figure out the social layer, recommendation engine, and just focus on getting rid of the tracker.  How?

I think they should create a tool where you can type in the SHA1 sum of any piece of content,* and it'll download it.  That's it.  Nothing more.  Think of what kind of interesting applications could come about if the recommendation engines and such didn't have to host torrent files, and didn't need to take on the risk of trackers!

* In practice it'd probably be a SHA1 sum of a list of SHA1 sums.

Because really, search engines like TPB only make sense when "what we have" is a small subset of "what you want", and when there is no general consensus on what a "good" copy of each thing is.  In this environment, you literally need to "search" for something you want.

But in practice, TPB has pretty much everything, and for each thing there's usually one version that pretty much everybody uses (ie, the version with all the seeds).  So there's usually little "searching" involved.  More often than not, just type in the name of what you want, pick the one with the highest number of seeds, and you're done.

Given this reality, it probably makes more sense to ditch the unstructured search interface and go to a structured "table of contents" indexing "all music" and "all movies", with the "best" version of each one given front and center.  Then you just click a "p2p://<sha1sum>" link and your client connects to the cloud and pulls it down.  (And all the recommendation engines would just layer atop that.)

Accordingly, TPB shouldn't copy Google: they should copy IMDB.  Create a comprehensive library of all content, and provide one recommended copy of each (or maybe a selection of encodings: iPod, HD, etc).

As for how to manage spam and such, again, embrace reality.  In theory, anybody could post a good copy of anything, and nobody is more trustworthy than any other.  In practice, there's usually one guy who is the uber-fan of a particular type of content, and that person posts all the good stuff.  Why not just explicitly recognize that uber-fan by making him moderator of the corresponding ThePirateIMDB wiki-like page for that band/TV-show/movie/etc.  Then you stop moderating content on a piece-by-piece basis, and start moderating on a curator-by-curator basis.  That uber-fan needn't be the one to post all good content (though in practice he probably would); others could recommend content to him and he'd sift through and find the good stuff.

Anyway, that's all just dreamy "what I'd do if I were a megapirate" talk.  Somebody's going to do it, and it probably won't be me.  I highlight it to make it clear that this is inevitable.  If this scenario frightens you, then your instinct is right: be afraid.  The future is coming, and if it's not to your liking, then now's the time for some deep introspection because there ain't nothing you can do to stop it.

The weak link in BitTorrent from a piracy perspective has always been the torrent sites.  They're the last centralized holdout vulnerable to attack from copyright enforcers (though those attacks have so far been futile).  Regardless, that vulnerability seems to be on the demise with the latest release of Tribler, which includes totally decentralized tracking ability.

I've long said copyright's days are numbered, and tools like this just make that number smaller and smaller.  Sure, copyright will still be enforceable on major customers like movie studios, satellite radio services, and other entities with a large financial and physical presence -- large enough to be worth defending, and worth attacking.

But Joe Plumber will be given an increasingly free hand to ignore copyright with impunity.  Whether that's morally right or wrong isn't the issue.  It's simply true, and more true every day.


In other news, I'm particularly interested in learning more about Tribler's "Give-To-Get" algorithm.  (The website is slammed right now, so I'll have to check it out later.)  I'm hoping/assuming it takes a less paranoid stand than the standard "tit-for-tat" algorithm BitTorrent employs, recognizing that the universe doesn't reset at the end of each download.

In short, if we share data via tit-for-tat, I only give you data if you also immediately give me data.  If you don't have any data for me, or if you give it to me slowly, then I'll withhold my data from you.  In a sense, data is like currency.

This is a brilliant model that allows for the protocol to succeed in scarce network conditions with different implementations: it protects each user from wasting data on users who don't respond in kind.

But it also makes downloads go unnecessarily slow in an abundant network situations because you can only download (on average) as fast as you can upload.  And because uploading is generally constrained to about a quarter your download speed, that means you can only generally download about 25% as fast as you could otherwise.

Now, BitTorrent gets around this with "seeds", who volunteer data without asking anything in return.  With enough seeds, anybody can download at full speed.  But seeds undermine the whole notion of tit-for-tat.

Indeed, the easy availability of seeds suggests that the whole assumption of tit-for-tat -- the scarce network environment -- is wrong.  Somehow, lots of users are more than willing to give away their bandwidth for free, without any obligation to do so.

Now, there are clever design decisions that encourage this: most torrent clients automatically begin seeding once you finish your download, some tracker sites monitor "seeding ratios" (the ratio of data uploaded to data downloaded), etc.

But the point is: despite there being no technical requirement for people to seed, people still do so, in huge numbers, and don't care.

Which brings me back to the original point: if this is "true" about the universe, then tit-for-tat is non-optimal.  It's like wearing a stillsuit in a rainforest.

So the question is: what *is* optimal.  And the answer is: upload when it's *cheap*, not when it's expensive.  Let me explain that:

Tit-for-tat makes you upload at the same time you download.  But the act of uploading actually makes you download slower.  Even worse, because downloading fast requires uploading fast, then the faster you download the more download capacity you're spending on uploading.  The upshot is even were it not for the asymmetric upload/download bandwidth ratio, tit-for-tat makes you upload when bandwidth is the most expensive.  Tit-for-tat takes a scarce bandwidth environment and makes it *worse*.

The alternative is to wait until the download is done and the upload later, when your network is idle (such as when you are watching the thing you just downloaded).  This way when you download, you download as fast as possible without wasting time uploading.  And when you upload, do it in a way that minimizes its impact upon the user who is volunteering the bandwidth.

Doing this, however, requires trust else people will download without ever uploading.  That trust is very difficult to enforce against people's wills.  (Even tit-for-tat suffers from BitThief problems.)

But the very fact that seeders are in such large supply in a tit-for-tat model shows that users are generally willing to donate their bandwidth voluntarily.  As such, even though there's no way to force people to upload, people still do it anyway.  If you make it more convenient to just "do the right thing" than try to fight the system, then people will just go with it and everybody wins.

Hopefully Tribler does this.  Once the website comes back, we'll see.

What should the movie studios do to avoid a fate similar to the major music labels?

The labels mined and salted the fields of digital music such that commercial success was impossible.  Their legacy will be intractable resistance to and wholesale destruction of the commercial music industry, effectively sending it back to the stone age.

What should the movie studios do differently?  To be certain, they're starting off on an equally self-destructive course, as this RealDVD episode shows.  But what other card do they have to play?

All I can think of is convenience and a superior experience.  For example, I know how to download pretty much any movie or TV show, but I still rent movies and series's all the time from Blockbuster just because it's way more convenient.  Likewise, I go to movies all the time because I am a sucker for the big screen.

The only meaningful asset the movie studios have is people don't absolutely despise their existence and wish them dead.  (The music labels were never so lucky.)  I wonder if they'll realize how valuable this asset is before it's lost, and if they realize how quixotically battling the future one RealDVD at a time earns them absolutely nothing while eroding the popular support they utterly depend upon to survive.

- David Barrett

PC World reports that Australia is considering a plan to scan for pirated music at border crossings, just one of many treats in a broader international treaty propping up the war on pirates, or citizens, or somebody.

Ignoring whether that plan makes any sense at all, how would it be done?  One way they could do this would be to switch to a "proof of payment" system, and use sampling for fast scans.  Basically, pick a random 10% of songs on the device, check their waveform fingerprints against some copyright database, and then verify that there is a digital signature embedded in the MP3's ID3 tag proving that the name of the customer who bought the song matches the name on the passport.

However, I don't really see this actually happening in any wide scale.  To make it workable too many things would need to happen, one of which is the music labels actually adopting digital purchases for real and then forcing all legit distributors to include information in each file.  It's not technically impossible, and would have been quite easy had they decided to do it in '98 when all the online merchants were begging for instructions and permission to make legit services.

Furthermore, the obvious response to this is to just put all pirated music in a hidden encrypted volume.  I'd expect somebody would come out with an application for "unlocked" iPhones that lets you enter a password to unlock the hidden volume, designed in such a way that without the right password it's impossible to know the hidden volume even exists.

The upshot is -- once again -- technical advantage goes to the pirates, as they can retool far faster than the TSA.  It would take years and years of complex negotiation on the part of a hundred corporations and government agencies, and it would all be rendered completely irrelevant by a simple, free iPhone application released by a nameless Russian programmer.

So once again, to anybody who's listening, give it up.  Copyright enforcement is and will be forever hopeless in this modern age.  Find another way to flourish.

-david barrett

PS: Just because it's fun to see your predictions validated, let me share an email I sent to a private mailing list a couple months back that seems strangely prescient.  (Though admittedly, only somewhat prescient because it's not that hard a leap to make.)

-------- Original Message --------
Subject: Re: An Essay Concerning MPAA Understanding of 'Making Available' in the P2P Context
Date: Thu, 26 Jun 2008 12:42:23 -0700
From: David Barrett <dbarrett@quinthar.com>

I'll take a stab at my own question and say "yes", but the shift will go from pursuing distributors to pursuing downloaders.  And I think they'll next try some sort of "proof of payment" scheme, such as used by public transportation:

In San Francisco, there are MUNI trains that you can board anywhere and get off anywhere; there's no physical requirement to buy a ticket. However, you're legally obligated to have one, and if you fail a spot inspection by an officer of the law, you'll pay serious fine.

I wonder if that's the model they will attempt next if "making available" fails.  Basically, all stores will move to individually-tagged songs and movies where proof of purchase is encoded in the content itself.   (This is impractical in the old world of physical media distribution, but becomes more feasible as we move to on-demand downloads).

One way to do this would be with watermarks: so long nobody has incentive to remove them, they'll stick around fine.  But then again, you could probably do it with just ID3 tags and digital signatures (a message "Bob has bought track <SHA1>" signed by Time Warner's public key would suffice).  Technically it's an easy problem to solve.

The problem will come in the audit: both how to audit the devices in question, and when to do it.

As for how, the challenge (as always) is to distinguish between content in the public domain and content you need permission from the copyright owner to have.  One possibility would be to build an opt-in waveform fingerprint of all copyrighted works that elect to participate in this proof of payment scheme.  This won't truly catch everything (and won't catch anything released before the scheme launched), but even if it catches only the new releases with some regularity, that starts to make an effective tool for general compliance enforcement.

So, auditors could conceivably have a device that has USB and iPod connectors that plug into basically anything, scan all content for waveform matches, confirms the file has a proof of payment certificate, and alerts if not.

Ok, so all this could technically be built by a sufficiently incented (or incensed?) party.  This brings us to the next question: when would the audit occur?

This is where it'd probably fail on constitutional grounds.  A scan under most circumstances would be "unreasonable search and seizure". But one place that is notoriously exempt: border control.  They can basically take anything and do anything for as long as it takes.

Granted, this cedes the vast majority of domestic piracy.  But their goal isn't to eliminate the potential for piracy; their goal is to make it such a pain that people still choose to buy.  If they first make it impossible to travel internationally without first cleansing all devices of pirated works, this will start to bite.  And after that, they'll find other excuses to audit devices: airport security for domestic flights?
PCI and SOX compliance audits?  Build auditing straight into the iPhone itself?

The big question in my mind is whether everybody just gives up on copyright before then and "just says no" to proof of payment and spot copyright checks.

By and large, society as a whole has already given up on copyright, as evidenced by overwhelming adoption of piracy.  It's possible that if pressed to make a decision that we'll simply refuse to pass any law that allows for reasonable enforcement.  Then businesses that depend on enforcement will die and get replaced with those that don't, and gradually the courts will limit the scope of copyright to where it can be realistically enforced.

Anyway, so I see a copyright-free (or copyright-very-limited) future as a legitimate possibility.  And society might just refuse to allow the proof-of-payment scheme to go into force.


So, let me conclude with my prediction: if "making available" fails (and if they truly accept this -- not necessarily a sure bet), then major copyright holders will marshal their forces and attempt to create a "proof of payment" system with enforcement starting at border crossings and gradually increasing from there.  This will trigger a showdown with society at large as it really begins to weigh how much it cares about copyrights, and the people who hold them.  And I think it's very possible that society decides the cost of copyright enforcement outweighs its benefit and essentially curtail copyright in all areas where it stopped making sense, long ago.

-david

Fantastic article in the New Yorker (via Slashdot) discussing the "Tragedy of the Anti-Commons" -- summarized as "The commons leads to overuse and destruction; the anticommons leads to underuse and waste."

The term has apparently has been around for a while, but that I hadn't heard it before.

Regardless, I'm particularly interested in the theory at the end for why the anti-commons goes underused, to everybody's detriment.  Basically, everybody over-estimates the value of their individual component, meaning the entire joint venture becomes more expensive to execute than it's actually worth.  Fascinating stuff.

- david barrett

There are always two responses to this sort of thing.

One class says "Geez, <prosecution> are idiots for not recognizing the potential for new revenue and partnering with YouTube!"

The other class says "Yep, YouTube is a criminal racket hiding behind a thin veneer of flimsy, untested law -- it's amazing they've gotten away with it for so long."

Granted, both could be right (they're not strictly contradictory).  But I tend to align more with the latter camp.

This doesn't mean I think YouTube is morally abject.  Rather, I think the law is stupid.  (Both the law they're guilty of breaking, and the law they use as a defense.)  But the law is the law, and it's frustrating to see YouTube profit from such blatant criminal activity** while so many others -- most of who were far more creative in either trying to comply with or circumvent the law -- were ground into dust.

- david barrett

** Yes, I realize the jury's out on what fraction of today's traffic is copyright infringing.  But there's little debate that YouTube's founding principle was massive copyright infringement, and only through a stroke of luck and the grace of time has managed to attract a sufficiently non-criminal userbase to maintain plausible deniability.