I really dislike the term "agile development".  I'm fine using "agile" to describe people or teams.  But using it to describe a methodology seems to completely miss the point.  You wouldn't say a gymnast uses an "agile technique".  You'd just say the gymnast "is agile".

So articles like "When Agile Projects Go Bad" sorta confuse and grate on me.

You'd never read an article titled "When Agile Gymnasts Fall" because it makes no sense; a gymnast who falls is -- by definition -- not very agile.  Similarly, an agile project that fails due to a lack of agility is rather paradoxical.  (Though there are many other ways to fail than by lacking agility.)

But this isn't just a problem of semantics.  Rather, I think this is symptomatic of a broader problem in the agile movement: it doesn't frickin' work.  To borrow from my good friend Lao Tze:

You can't learn agility by reading and following.  You become agile by doing, failing, changing, and doing again.  The most agile people I know read the fewest books on it.  Similarly, I don't know anybody who's seriously studied the subject of agile development techniques gotten the least bit better.

So here's my advice: if you want to be agile, put down the book and just start making it up as you go.  If what you're doing isn't working, try something different.  If what you're doing works, try cutting out a step and see if it still works -- or even works better.  Repeat.

You're certain to make wrong steps.  You're certain to encounter failure -- indeed, failure will likely be your steady state.

But eventually you'll figure it out, and every once in a while it'll work out great.

Welcome to the world of agility.  No reading required.

- David Barrett

I know there's a dearth of real-world data on piracy, but here's a dash: the Pirate Bay recently passed the 25M mark in terms of simultaneous peers.  Here's the data itself, from TorrentFreak:

    Date            Peers (M)
    11/01/2006      1
    11/01/2007      6
    09/21/2008      15
    11/01/2008      20
    11/15/2008      25

That's interesting, but it looks much more compelling graphically:



So the rate of real pirates seen by The Pirate Bay has increased by 25x in just over 2 years.  (As for how meaningful this data is, it's suggested that TPB tracks about 50% of all torrents.)

Anybody care to guess whether that 20:1 ratio of illegal to legal downloads ratio is going?  Might it be more like 100:1 today?  The growth of piracy over legal downloads is just staggering.

-david

So I finally made it through to the Tribler website and I think it's *almost* really cool.  Don't get me wrong, I installed the app and it seems to do interesting stuff.  But I think they're biting off more than they can chew.  Basically, I'd layer the content-acquisition experience as follows:

Social: The experience of sharing and learning about new content
Index: Determining who has a given piece of content
Transport: Getting content from somebody else

Tribler tries to vertically integrate the entire stack, and they do a decent job, but it's just too much for one application.  It's like coming out with Prodigy today and trying to complete with "The Internet".  If Tribler came out before ThePirateBay then it'd be something.  But then it'd be called Kazaa.

No, instead I think they should do something less.  Indeed, the whole brilliance of BitTorrent was that it *didn't* do it all.  BitTorrent does nothing but standardize transport, which enabled a huge diversity on the upper layers.  Doing *less* than Kazaa is what made BitTorrent succeed.

So if BitTorrent is on one side, and Kazaa on the other, what should Tribler do?  I'd say stick with the old saying "only innovate one thing at a time" and just integrate the index into the transport layer, and then call it day.  Leave ThePirateBay and everyone else to figure out the social layer, recommendation engine, and just focus on getting rid of the tracker.  How?

I think they should create a tool where you can type in the SHA1 sum of any piece of content,* and it'll download it.  That's it.  Nothing more.  Think of what kind of interesting applications could come about if the recommendation engines and such didn't have to host torrent files, and didn't need to take on the risk of trackers!

* In practice it'd probably be a SHA1 sum of a list of SHA1 sums.

Because really, search engines like TPB only make sense when "what we have" is a small subset of "what you want", and when there is no general consensus on what a "good" copy of each thing is.  In this environment, you literally need to "search" for something you want.

But in practice, TPB has pretty much everything, and for each thing there's usually one version that pretty much everybody uses (ie, the version with all the seeds).  So there's usually little "searching" involved.  More often than not, just type in the name of what you want, pick the one with the highest number of seeds, and you're done.

Given this reality, it probably makes more sense to ditch the unstructured search interface and go to a structured "table of contents" indexing "all music" and "all movies", with the "best" version of each one given front and center.  Then you just click a "p2p://<sha1sum>" link and your client connects to the cloud and pulls it down.  (And all the recommendation engines would just layer atop that.)

Accordingly, TPB shouldn't copy Google: they should copy IMDB.  Create a comprehensive library of all content, and provide one recommended copy of each (or maybe a selection of encodings: iPod, HD, etc).

As for how to manage spam and such, again, embrace reality.  In theory, anybody could post a good copy of anything, and nobody is more trustworthy than any other.  In practice, there's usually one guy who is the uber-fan of a particular type of content, and that person posts all the good stuff.  Why not just explicitly recognize that uber-fan by making him moderator of the corresponding ThePirateIMDB wiki-like page for that band/TV-show/movie/etc.  Then you stop moderating content on a piece-by-piece basis, and start moderating on a curator-by-curator basis.  That uber-fan needn't be the one to post all good content (though in practice he probably would); others could recommend content to him and he'd sift through and find the good stuff.

Anyway, that's all just dreamy "what I'd do if I were a megapirate" talk.  Somebody's going to do it, and it probably won't be me.  I highlight it to make it clear that this is inevitable.  If this scenario frightens you, then your instinct is right: be afraid.  The future is coming, and if it's not to your liking, then now's the time for some deep introspection because there ain't nothing you can do to stop it.

The weak link in BitTorrent from a piracy perspective has always been the torrent sites.  They're the last centralized holdout vulnerable to attack from copyright enforcers (though those attacks have so far been futile).  Regardless, that vulnerability seems to be on the demise with the latest release of Tribler, which includes totally decentralized tracking ability.

I've long said copyright's days are numbered, and tools like this just make that number smaller and smaller.  Sure, copyright will still be enforceable on major customers like movie studios, satellite radio services, and other entities with a large financial and physical presence -- large enough to be worth defending, and worth attacking.

But Joe Plumber will be given an increasingly free hand to ignore copyright with impunity.  Whether that's morally right or wrong isn't the issue.  It's simply true, and more true every day.


In other news, I'm particularly interested in learning more about Tribler's "Give-To-Get" algorithm.  (The website is slammed right now, so I'll have to check it out later.)  I'm hoping/assuming it takes a less paranoid stand than the standard "tit-for-tat" algorithm BitTorrent employs, recognizing that the universe doesn't reset at the end of each download.

In short, if we share data via tit-for-tat, I only give you data if you also immediately give me data.  If you don't have any data for me, or if you give it to me slowly, then I'll withhold my data from you.  In a sense, data is like currency.

This is a brilliant model that allows for the protocol to succeed in scarce network conditions with different implementations: it protects each user from wasting data on users who don't respond in kind.

But it also makes downloads go unnecessarily slow in an abundant network situations because you can only download (on average) as fast as you can upload.  And because uploading is generally constrained to about a quarter your download speed, that means you can only generally download about 25% as fast as you could otherwise.

Now, BitTorrent gets around this with "seeds", who volunteer data without asking anything in return.  With enough seeds, anybody can download at full speed.  But seeds undermine the whole notion of tit-for-tat.

Indeed, the easy availability of seeds suggests that the whole assumption of tit-for-tat -- the scarce network environment -- is wrong.  Somehow, lots of users are more than willing to give away their bandwidth for free, without any obligation to do so.

Now, there are clever design decisions that encourage this: most torrent clients automatically begin seeding once you finish your download, some tracker sites monitor "seeding ratios" (the ratio of data uploaded to data downloaded), etc.

But the point is: despite there being no technical requirement for people to seed, people still do so, in huge numbers, and don't care.

Which brings me back to the original point: if this is "true" about the universe, then tit-for-tat is non-optimal.  It's like wearing a stillsuit in a rainforest.

So the question is: what *is* optimal.  And the answer is: upload when it's *cheap*, not when it's expensive.  Let me explain that:

Tit-for-tat makes you upload at the same time you download.  But the act of uploading actually makes you download slower.  Even worse, because downloading fast requires uploading fast, then the faster you download the more download capacity you're spending on uploading.  The upshot is even were it not for the asymmetric upload/download bandwidth ratio, tit-for-tat makes you upload when bandwidth is the most expensive.  Tit-for-tat takes a scarce bandwidth environment and makes it *worse*.

The alternative is to wait until the download is done and the upload later, when your network is idle (such as when you are watching the thing you just downloaded).  This way when you download, you download as fast as possible without wasting time uploading.  And when you upload, do it in a way that minimizes its impact upon the user who is volunteering the bandwidth.

Doing this, however, requires trust else people will download without ever uploading.  That trust is very difficult to enforce against people's wills.  (Even tit-for-tat suffers from BitThief problems.)

But the very fact that seeders are in such large supply in a tit-for-tat model shows that users are generally willing to donate their bandwidth voluntarily.  As such, even though there's no way to force people to upload, people still do it anyway.  If you make it more convenient to just "do the right thing" than try to fight the system, then people will just go with it and everybody wins.

Hopefully Tribler does this.  Once the website comes back, we'll see.

I'm not pro-suicide.  But $40-$50 million dollars + $78K/year to build a net under the Golden Gate Bridge in order to dissuade just a few dozen jumper a year seems outrageous, on so many fronts.

First, anybody who actually does jump is probably pretty serious about killing themselves -- serious enough that they'll find some other way.  So probably the most absurd part is it being a completely stupid and pointless plan on its face that will probably end up saving zero lives.

But ignoring that -- after all, I'm willing to endorse symbolic plans on occasion -- the price for this meaningless gesture is astronomical.  $40 - $50 *million* dollars?  To encourage only 39 jumpers a year to go somewhere else?  Who can possibly suggest it's a wise expenditure of money, especially in this economic climate, to spend over a *million dollars* to stop just *one* jump a year?

To put that in perspective, if we kept that same money as cash, we could spend over $100,000 per jumper per year for the next century.  We could hire 50 full-time-people to just stand there, 24/7, and watch the bridge -- perhaps talking down anybody who looks like they might jump -- until 2108.  Even just investing $50M dollars at a 5% interest rate would earn $6.5 *billion* dollars in 100 years. 

Even if it were "only" the $78,000 per year maintenance fee (that's right, once built, it needs to be maintained), that's like $2000 per jumper per year.  Even that "paltry" amount could be better spent saving actual lives, or even just hiring another full-time member at a suicide prevention line.

So the cost is outrageous and simply indefensible, especially given it will completely fail to accomplish its objective.  But on top of this, the Golden Gate Bridge is a historic landmark that probably brings in billions of dollars a year in tourism to San Francisco.  We're seriously going to be some huge frickin' ugly net under it?  What kind of effect will that have on tourism or even our international reputation?

Who is in charge of this boondoggle of a plan, and is there any time to breathe sense into the process?  I'm down with spending money to save lives.  But this is just such a ridiculous waste of money it's infuriating.

-david

What should the movie studios do to avoid a fate similar to the major music labels?

The labels mined and salted the fields of digital music such that commercial success was impossible.  Their legacy will be intractable resistance to and wholesale destruction of the commercial music industry, effectively sending it back to the stone age.

What should the movie studios do differently?  To be certain, they're starting off on an equally self-destructive course, as this RealDVD episode shows.  But what other card do they have to play?

All I can think of is convenience and a superior experience.  For example, I know how to download pretty much any movie or TV show, but I still rent movies and series's all the time from Blockbuster just because it's way more convenient.  Likewise, I go to movies all the time because I am a sucker for the big screen.

The only meaningful asset the movie studios have is people don't absolutely despise their existence and wish them dead.  (The music labels were never so lucky.)  I wonder if they'll realize how valuable this asset is before it's lost, and if they realize how quixotically battling the future one RealDVD at a time earns them absolutely nothing while eroding the popular support they utterly depend upon to survive.

- David Barrett

Isn't it really surprising how much people hate this $700B bailout bill?  

I read a report there was near unanimous opposition among constituents -- so much opposition that websites were crashing under the volume.  I thought this was probably bull, but when I went to call Barbara Boxer I found her voicemail full.  On her website there's a special note that the contact form might not work due to high volume.  Her alternate number is full.  Diane Feinstein's number is busy.

(Here's a helpful page showing California Senator contact info.)

Is this level of true grassroots opposition unprecedented, especially given that both Republican and Democratic Party leaders support it unequivocally?

Personally, I agree with the masses.  America is going bankrupt and I'm much more concerned about another $700B of debt -- especially when I have zero confidence it will actually accomplish its intent of rescuing the economy -- than waiting to see what happens and letting the market sort itself out.  And now that they're trying to sweeten the deal by trowing in tax cuts?  Have we completely lost our minds when it comes to fiscal responsibility?

Furthermore, the defense is just absurd.  Granting that there are probably really good arguments for it, given that this whole episode was triggered by irresponsible lending, can't we find some other defense than "this is needed to let people borrow money to buy homes"?

Regardless, it sounds like the masses' opposition might have tapered off given the stock market crash, so who knows.  Maybe we'll get bailed out, whether we like it or not.

-david

The other day I read a post on a mailing list lambasting the current state of the union (specifically how both candidates are shirking their jobs to run for office) and saying, essentially "it's not my fault, I voted Libertarian".  I've heard that a bunch of times in a variety of forms, so I had to respond as follows:

"If you're advocating a policy change to prevent active senators, legislators, governors, and so forth from campaigning while on the job, I think you'd have many supporters from all sides.  We need more of that constructive discussion.

But it's not helpful to bash the status quo and then claim you're not responsible merely because your party consistently loses at the polls.

If you're a US citizen, then you're responsible for the result of our political system, whether or not your guy won.  Voting for the loser doesn't excuse you from the results of the process.  It just means you're doing a shitty job."

We're all equal participants in this process.  We all share equal responsibility for its results, both the good *and* the bad.  if you dislike like those results, don't vent: rally some friends and make a difference.  Because if you don't, somebody else who thinks otherwise will just rally their friends, harder.

So I'm debugging this SOAP protocol and having a certain problem: me and a vendor are submitting the "exact same" request but getting different responses. Furthermore, each of us repeatedly get the same response to our requests, but never get each other's.

At this point I realize that we have different meanings of "exact same". In my mind, "same" means "the bytes that go over the wire are identical". In their mind, "same" means "the objects being serialized have the same data". The difference there, of course, is the manner of serialization. Different request serializations produce different responses. Same != Same.

This made me realize that we all have different notions of how the internet works. I'm a low-level guy so I think in terms of bytes, whereas others think in terms of higher level protocols. I think there are benefits and detriments to thinking on every layer, but the most important thing is to identify when you're thinking on *different* layers.

Furthermore, it's caused me to wonder what I'm missing out on by thinking of the internet as a series UDP/TCP flows. What magical wonders are happening at the IP layer that I'm just overlooking? How does BGP *really* work? What are the actual differences between 802.11 a-g? I have no idea.

But I bet there's going to be a lot of exciting developments happening at these lower layers as programmable radios become commonplace and the average Joe can suddenly whip up a new wireless protocol from scratch.

Similarly, what's going on at those higher layers? I've heard of microformats and semantic data and agent programming and the like. It all sounds dreamy. But maybe I should be considering them in a more practical sense?

Regardless, back to the present. It's always an interesting to discover that the common ground you think you share isn't so common after all. My notion of the internet is not necessarily your notion, so beware.

Expensify isn't so much a labor of love as it is a retaliation against paper expense reports.

Don't get me wrong: making business purchases doesn't phase me.  I'm fine buying tickets, renting cars, booking hotels, and generally getting things done on my own credit card.  But the process of getting paid back is just so excruciating.

I'm not sure which is the worst: keeping track of the receipts, typing it all in, waiting to receive the check, or actually depositing it once I get it.  The whole experience is so bad it blurs together into a general morass of pain.  It's so bad I find myself just paying for things out of pocket because that's preferable to the torturous process of getting reimbursed.

This has been the case for years, and as I clawed my way up the startup circuit, I always rather assumed the big companies had some magical solution that made it all easy.  Well, they don't.  It's horrible from top to bottom, and what few corporate card solutions exist are targeted squarely at the big guys.  I've got news for you: there are 25 million businesses in the US today, and 24 million of those have under 20 employees.  That means 97% of the market is has been neatly overlooked, leaving the rest of us in expense report hell.


Well, I'm proud to say that all that ends now.  Expensify is the corporate card for everyone else, the expense report service that not only pays you back faster, but pays you back *more* -- and does so without the grueling pain we've all come to associate with the words "expense report".

In case you don't already know, Expensify uses electronic payment cards to enable one-click expense reports.  Every purchase you make with the card we give you is billed back to your regular credit card -- there's no new monthly bill, and you keep all your frequent flier miles.  Come back to the site at any time to print out full expense reports (including receipts: just scan them in with our iPhone app or email them to receipts@expensify.com using any cameraphone), or type in your manager's email address and we'll mail them a PDF and bill them for the amount of the expense report, crediting it back to your credit card.  No checks to cash, no days to wait, we'll just instantly pay off your credit card when your boss approves the report.

Anyway, you might have heard we launched last week at the TechCrunch 50 and Wow.  What a response.  We were cautiously optimistic that people would like it -- we were unprepared for how much people would absolutely *love* it.  We pitched it over 150 times, and everyone was ecstatic.  One guy even gave me a hug as thanks for how much time we were going to save him.  I think he had a tear in his eye.

All this enthusiasm has really highlighted to us just how big a problem we're solving, and how important it is that our launch go smooth.  So as much as it pains me, we've decided to slow things down a bit and put some extra testing and polish into the site, and to send out our cards in smaller batches than initially anticipated.  We'll still send them out on a first-come first-serve basis, and once we're through the backlog our plan is still to get cards to your door within a week.  But we're going to take our time with these first batches to ensure the highest quality experience possible.

That said, we want to thank you, the TechCrunch crew, and most importantly all the many, many people who spoke with us at the TechCrunch 50 and gave us such kind words of support and encouragement -- not to mention voted us 2nd place winner of the Demo Pit over a hundred other companies.  And what better way to give thanks than with cold hard cash!

That's right, from this moment on, the next 1000 users who sign up will get $1000 in free purchases.  That's a cool million dollars of Expensify service we're giving away for free to you, the early adopters, the TechCrunch readers, and the fledgling group of Expensify fans who make it all possible.  So sign up today and get in line for your Expensify card and $1000 of zero-surcharge purchases.  Your card will be in your mailbox before you know it, and your dread of expense reporting will soon be a distant memory!

Thanks again to Jason Calacanis, Michael Arrington, and the entire TechCrunch 50 crew for putting together a fantastic venue and the best launching pad that anybody could ask for.  These guys did a fantastic job, under difficult circumstances, and I can't wait to see what they cook up next year.  With any luck, Expensify will be there!

- David Barrett, Founder, Expensify

Well, the word is out: Expensify is live and taking orders for our one-click expense report service.  I'll have more to say in a bit, but for the moment I'll let Travis Kalanick (our advisor) do the talking:

"Expensify is launching at TC50!

Who here has a bag of stale receipts sitting in their closet?  Receipts that long, long ago should have made it into some expense report. . . of course to get there, you would have had to organize that big bag of receipts in chronological order, affixed and taped each of them to a separate blank sheet of paper, gotten some excel sheet or web-form and done a few hours of data entry.  Of course, you’d also have to remember what was discussed at that steak dinner in Denver (it was with customers. . .yeah, I’m sure it was), and then list their full names and titles.  The list of headaches stretches for miles. . .Taxi receipts not filled out, separating room charges from room service and hotel Internet, old faded receipts that are illegible. .. the list goes on and on.

So I think we all would agree that there is one word that is synonymous with doing expense reports. .. PAIN. 

So much pain that it takes up to 45 minutes per $1000 of expense reporting.  For employees with expense-prone job descriptions (there are 50 million of you in the US: salespeople, office managers, small-business owners) that comes to 30-40 hours a year.  That’s a week’s worth of vacation, or a week more of actual selling to hit your quota.

The pain for many of the less expense-disciplined among us, hits us in our pocket book.  Some estimates are that 5% or more of all legitimate expenses never get reimbursed because of receipts that get lost, and expense reports that don’t get filed (referring to that bag of wilting receipts above).

Expensify wants to change that

Expensify’s Company Mission:  Empower small-business employees, independent contractors and sole proprietors with easy tools for PAINLESS expense reporting.

Expensify accomplishes this with 3 main Expensify components:

1) Expensify expense card – a card that you use to make expense purchases.  You top-up Expensify card with your existing credit card. The card makes expense categorization a breeze, and allows Expensify to automatically fill out expense reports.

2) Receipt capture and upload – take pictures of your receipts and upload with either Expensify’s iPhone application or with simple email attachments from your phone or PDA.

3) Expensify Dashboard – Expensify automatically associates the receipt images with the expense card entries.  You’re only a few clicks away from a completed expense report.  Submit digital expense reports to your boss or client (via email) or print out and send physical expense report.

So now you’re saying, this is all way too easy.  Expense reports done with only a few clicks, submitted before I even get back to the office from a trip.  No more lost receipts or expense report hell??!!  How much does Expensify cost?

Identical to PayPal pricing on an existing credit card, Expensify charges 3% of transactions made to the Expensify expense card.

How can you justify Expensify cost??  If Expensify saves even one receipt in thirty from getting lost in the shuffle, the extra reimbursements means Expensify pays for itself!  We also know that time is money, and the time you save not dealing with expense reports should add to your overall quality of life

And remember, Expensify charges ARE reimbursable expenses!!

So how do I get started?

Step 1 – Register at Expensify.com
- Enter email address and password
- Verification email sent to you
- Sign up for Expensify expense card

Step 2 – Set up iPhone/PDA for receipt image capture
- iPhone users: Install iPhone application for quick uploading of your receipt images
- Other phones/pda’s: Email photos to receipts@expensify.com

Step 3 – Impatiently wait for your Expensify expense card to be mailed to you
- Should be delivered in one week

You’ve now been EXPENSIFY’D!!  Enjoy easy no-hassle expense reporting!

Thanks for checking us out.  Give Expensify a shot, let us know what you think, and don’t hold back. .. feedback for the product, venting over previous expense report nightmares, just sound off. .. . and thanks for getting EXPENSIFY’D!!

Travis Kalanick
Expensify Advisor and happy customer"

A great quote from Travis: "David thought it was a good idea, but when I started digging into the data, I determined it was a *great* idea."  Thanks for the intro Travis, I'll follow up in a bit once I have some free time -- the next flood of TechCrunchers are on their way!

Adam Frisk is an awesome P2P engineer and a major contributor to both Limewire and now Littleshoot.  He's trying to put together a panel at SXSW -- go sign up and vote for him today!

PC World reports that Australia is considering a plan to scan for pirated music at border crossings, just one of many treats in a broader international treaty propping up the war on pirates, or citizens, or somebody.

Ignoring whether that plan makes any sense at all, how would it be done?  One way they could do this would be to switch to a "proof of payment" system, and use sampling for fast scans.  Basically, pick a random 10% of songs on the device, check their waveform fingerprints against some copyright database, and then verify that there is a digital signature embedded in the MP3's ID3 tag proving that the name of the customer who bought the song matches the name on the passport.

However, I don't really see this actually happening in any wide scale.  To make it workable too many things would need to happen, one of which is the music labels actually adopting digital purchases for real and then forcing all legit distributors to include information in each file.  It's not technically impossible, and would have been quite easy had they decided to do it in '98 when all the online merchants were begging for instructions and permission to make legit services.

Furthermore, the obvious response to this is to just put all pirated music in a hidden encrypted volume.  I'd expect somebody would come out with an application for "unlocked" iPhones that lets you enter a password to unlock the hidden volume, designed in such a way that without the right password it's impossible to know the hidden volume even exists.

The upshot is -- once again -- technical advantage goes to the pirates, as they can retool far faster than the TSA.  It would take years and years of complex negotiation on the part of a hundred corporations and government agencies, and it would all be rendered completely irrelevant by a simple, free iPhone application released by a nameless Russian programmer.

So once again, to anybody who's listening, give it up.  Copyright enforcement is and will be forever hopeless in this modern age.  Find another way to flourish.

-david barrett

PS: Just because it's fun to see your predictions validated, let me share an email I sent to a private mailing list a couple months back that seems strangely prescient.  (Though admittedly, only somewhat prescient because it's not that hard a leap to make.)

-------- Original Message --------
Subject: Re: An Essay Concerning MPAA Understanding of 'Making Available' in the P2P Context
Date: Thu, 26 Jun 2008 12:42:23 -0700
From: David Barrett <dbarrett@quinthar.com>

I'll take a stab at my own question and say "yes", but the shift will go from pursuing distributors to pursuing downloaders.  And I think they'll next try some sort of "proof of payment" scheme, such as used by public transportation:

In San Francisco, there are MUNI trains that you can board anywhere and get off anywhere; there's no physical requirement to buy a ticket. However, you're legally obligated to have one, and if you fail a spot inspection by an officer of the law, you'll pay serious fine.

I wonder if that's the model they will attempt next if "making available" fails.  Basically, all stores will move to individually-tagged songs and movies where proof of purchase is encoded in the content itself.   (This is impractical in the old world of physical media distribution, but becomes more feasible as we move to on-demand downloads).

One way to do this would be with watermarks: so long nobody has incentive to remove them, they'll stick around fine.  But then again, you could probably do it with just ID3 tags and digital signatures (a message "Bob has bought track <SHA1>" signed by Time Warner's public key would suffice).  Technically it's an easy problem to solve.

The problem will come in the audit: both how to audit the devices in question, and when to do it.

As for how, the challenge (as always) is to distinguish between content in the public domain and content you need permission from the copyright owner to have.  One possibility would be to build an opt-in waveform fingerprint of all copyrighted works that elect to participate in this proof of payment scheme.  This won't truly catch everything (and won't catch anything released before the scheme launched), but even if it catches only the new releases with some regularity, that starts to make an effective tool for general compliance enforcement.

So, auditors could conceivably have a device that has USB and iPod connectors that plug into basically anything, scan all content for waveform matches, confirms the file has a proof of payment certificate, and alerts if not.

Ok, so all this could technically be built by a sufficiently incented (or incensed?) party.  This brings us to the next question: when would the audit occur?

This is where it'd probably fail on constitutional grounds.  A scan under most circumstances would be "unreasonable search and seizure". But one place that is notoriously exempt: border control.  They can basically take anything and do anything for as long as it takes.

Granted, this cedes the vast majority of domestic piracy.  But their goal isn't to eliminate the potential for piracy; their goal is to make it such a pain that people still choose to buy.  If they first make it impossible to travel internationally without first cleansing all devices of pirated works, this will start to bite.  And after that, they'll find other excuses to audit devices: airport security for domestic flights?
PCI and SOX compliance audits?  Build auditing straight into the iPhone itself?

The big question in my mind is whether everybody just gives up on copyright before then and "just says no" to proof of payment and spot copyright checks.

By and large, society as a whole has already given up on copyright, as evidenced by overwhelming adoption of piracy.  It's possible that if pressed to make a decision that we'll simply refuse to pass any law that allows for reasonable enforcement.  Then businesses that depend on enforcement will die and get replaced with those that don't, and gradually the courts will limit the scope of copyright to where it can be realistically enforced.

Anyway, so I see a copyright-free (or copyright-very-limited) future as a legitimate possibility.  And society might just refuse to allow the proof-of-payment scheme to go into force.


So, let me conclude with my prediction: if "making available" fails (and if they truly accept this -- not necessarily a sure bet), then major copyright holders will marshal their forces and attempt to create a "proof of payment" system with enforcement starting at border crossings and gradually increasing from there.  This will trigger a showdown with society at large as it really begins to weigh how much it cares about copyrights, and the people who hold them.  And I think it's very possible that society decides the cost of copyright enforcement outweighs its benefit and essentially curtail copyright in all areas where it stopped making sense, long ago.

-david

Poor Pandora, and we loved you so.  (And by "we" I mean "they", as I wasn't a user.)  Regardless, I wonder if there's an inevitable cycle at play:

1) Business tries to do it legit
2) Business goes bankrupt due to impossible pricing
3) Pirate does it the easy way
4) Pirate gets sued to oblivion
5) Pirate does it the hard way.
6) ... it's free for the rest of eternity

If so, when it comes to web radio, it seems we're passing stage 2.  Next up should be a round of central pirate stations -- essentially large-scale shoutcast installations -- which briefly flourish followed by being wiped out.  The third wave should come in a couple years -- maybe some sort of centralized preference engine tied to decentralized streaming straight from trackerless torrents?  Sounds like a fun project!

- david barrett

Fantastic article in the New Yorker (via Slashdot) discussing the "Tragedy of the Anti-Commons" -- summarized as "The commons leads to overuse and destruction; the anticommons leads to underuse and waste."

The term has apparently has been around for a while, but that I hadn't heard it before.

Regardless, I'm particularly interested in the theory at the end for why the anti-commons goes underused, to everybody's detriment.  Basically, everybody over-estimates the value of their individual component, meaning the entire joint venture becomes more expensive to execute than it's actually worth.  Fascinating stuff.

- david barrett

Saw this article on the ease of framing arbitrary computers/users as pirates and I immediately thought: they should identify the IPs of bunch of congressmen, RIAA members, judges, and reporters and flood them with fake DMCA takedown notices.  (Or, rather, real notices for  fake downloads.)  What better way to get the attention of your representative than with a frivolous lawsuit?

- david barrett

There are always two responses to this sort of thing.

One class says "Geez, <prosecution> are idiots for not recognizing the potential for new revenue and partnering with YouTube!"

The other class says "Yep, YouTube is a criminal racket hiding behind a thin veneer of flimsy, untested law -- it's amazing they've gotten away with it for so long."

Granted, both could be right (they're not strictly contradictory).  But I tend to align more with the latter camp.

This doesn't mean I think YouTube is morally abject.  Rather, I think the law is stupid.  (Both the law they're guilty of breaking, and the law they use as a defense.)  But the law is the law, and it's frustrating to see YouTube profit from such blatant criminal activity** while so many others -- most of who were far more creative in either trying to comply with or circumvent the law -- were ground into dust.

- david barrett

** Yes, I realize the jury's out on what fraction of today's traffic is copyright infringing.  But there's little debate that YouTube's founding principle was massive copyright infringement, and only through a stroke of luck and the grace of time has managed to attract a sufficiently non-criminal userbase to maintain plausible deniability.

Curtis, my top source of scoops, pointed me to this breathless review of Sun's new transactional memory hardware, making bold predictions that Sun will -- and I quote: "horsefuck the database world".

But unless I'm really missing something, how does this help?  A database's bottleneck is not thread synchronization, it's atomic writing to disk.  Sure, transaction management of in-memory data is costly.  But not nearly as costly as writing updates to disk in an infallible manner.

Maybe there are specific database scenarios that really benefit from this.  But a read-heavy scenario doesn't have a lot of transactional overhead, and a write-heavy scenario will be bound by disk write performance.

So again, where does this help?  I must be totally missing something.  Can you help steer me straight?

-david barrett

Curtis pointed me to this interesting article questioning the hype surrounding distributed version control systems (DVCS -- ya, I hadn't heard the acronym before either).

I generally agree with him, but some of the comments were especially illuminating.  In particular, I like the comment that it turns the relationship around: rather than a master "pushing" changes out to the masses, everybody has the option of "pulling" from whoever they like.  I can see how this is particularly valuable in an open source environment without strong leadership, though less so in a commercial environment or an open-source community with a strong "mainline" (eg, Firefox).

The only feature that jumps out to me is offline commit, merely because I spend a lot of time offline (or did, until I got a laptop with built-in Sprint wireless broadband -- kick ass!).  And the proposed "waypoint" addition to SVN would get me 80% there.  (The other 20% would be sharing waypoints with other users, but in my experience that's an incredibly rare operation.)

But regarding the claims that DVCS is somehow fundamentally easier to use or less prone to weird boundary conditions (eg, "Git rocks because I hate it when I delete a modified uncommitted file using the OS after renaming the parent directory a prime number of times in a row"), I'm skeptical.  Every VCS has dark corners where none dare tread.

Similarly, the claims "it's hard to set up a SVN repository" are pretty weak: cvsdude?  rsync.net?  Or even SourceForge?  Likewise, setting up a local repository is kinda missing the point in a world of cloud computing: whether you like it or not, your laptop *will* break and your hard drive *will* fail.  Anything worth version controlling is worth backing up remotely, and SVN is as good a tool as any for that.

And finally, maybe I'm just really missing something, but I just don't get the value of extreme branch/merge.  I don't feel its complexity is what's preventing me from doing it -- I just never feel the need.  Indeed, it seems to go against the "continuous integration" ethos of agile development: I'd much rather have a bunch of somewhat-broken code prematurely integrated than somewhat-broken code integrated too late at the very end.

Granted this is probably due to my history of working with small teams on tight codebases (where everybody constantly affects everyone else): I can imagine how branching/merging becomes more valuable as team size and developer-decoupling increases.  And granted, I only very rarely submit or accept patches from contributors without direct commit access.

But all told, I agree with the B-List that the whole thing seems over-hyped.  It's a tool.  There are lots others.  Pick whichever one you like and get to work.

-david barrett

One critique of my previous post was that it doesn't account for streaming, such as from YouTube and MySpace.  Ok, here goes:

First, to clarify, YouTube music is generally unlicensed and posted illegally, so it should probably go in the pirate column.  (The DMCA gives protection to YouTube so long as it removes pirated material, but it doesn't change the fact that it's pirated in the first place.)

MySpace is a great example of legitimate streaming, though.

So even limiting the discussion to just streaming, I wonder about the ratio of pirated tracks streamed from DMCA-protected providers, versus legit tracks from MySpace/webcasting/etc.  Do you have any data on this?

PC World suggests over a *billion* videos per day on YouTube.  Let's say, what, half of those include unlicensed songs that aren't protected by fair use?  25%?  Even at 10%, that means 100M pirated songs per day streamed from YouTube.

Now, how many legit songs are streamed from MySpace?  Somewhere I saw 4.5B daily page views.  How many of those have songs on them?  I really have no idea -- I'm not a MySpace user.  I'm going to guess 25%.  Furthermore, my understanding is every page automatically starts playing music, so that would suggest like 1.1B legitimate songs started every day from MySpace -- dwarfing YouTube.  Then again, how many of those pages actually play the whole song, rather than playing a few seconds before the next click?  Again, no real idea, but I'd wager there's a lot of clicking going on, to the tune of 1 in 10 songs started actually being listened to for real.  That would put us down to about 110M legitimate streams from MySpace a day, or roughly on par with YouTube's piracy.  (In YouTube's defense, I wager most videos are actually played to completion.)

Oh, and to put it into context, my previous estimates put the number of songs bought from iTunes at 5.8M per day, suggesting about 116M downloaded per day from pirate networks.  So the number of songs *streamed* per day is in the same ballpark as number of songs *downloaded*, despite the downloaded songs presumably being played again and again.


Anyway, this is all guesswork -- I'd love real data.  But by my guesses I'd say piracy accounts for 95% of music downloads and maybe 25-50% of streaming downloads.  Furthermore, the download market (which piracy unquestionably dominates) results in far more actual songs listened to.

If we said that Toyota accounted for 95% of cars and 25-50% of trucks (with cars outnumbering trucks 10:1), we'd probably say they dominated the auto industry.  And thus I still think it's pretty safe to say piracy dominates all online music, streaming and downloaded alike.

-david

This was my response to a post about the oddity of Yahoo's latest of a string of failures in music players and services, comparing it to the broader ecosystem of the music industry:

"I realize it's beating a dead horse, but you left out the largest, most vibrant part of the "ecosystem", the only part worth discussing: piracy.

I still don't understand why -- frankly -- anybody cares what Yahoo and Napster and even iTunes do.  They're a largely irrelevant footnote, a strange march of impossibly flawed products targeting tiny niches of occasionally-curious users and manic well-wishers.  That's like discussing operating systems exclusively in terms of Minix and OS/360, while occasionally remarking "oh, ya, and there's this thing called Windows, but whatever."  **

And in this broader context, it doesn't seem strange at all that Yahoo's latest initiative failed.  Just like every other of its kind.  The world has very, very clearly said it doesn't want those, at every possible opportunity.  History books will remark "At the turn of the 21st century there were a series of attempts to commercialize digital content by enforcing an antiquated notion called 'copyright' -- the right to control who could make copies -- all of which obviously failed."

If there's anything we "need to accept" it's that piracy is the biggest game in town, has been from the start, and will probably be forever.

Piracy *is* online music, and the only lasting success anybody is going to see in this industry is by augmenting or avoiding it.  But not by competing head on.
 
-david

** OS/360 is the iTunes of the bunch: very relevant and commercially successful to a small demographic.  Indeed its success is precisely due to a recognition of not being for everyone."

The first internet bubble popped largely because all business models failed except for ad selling.  Is it possible that the last stalwart hope is itself doomed?

TechCrunch reports that Lookery, a company specializing in selling ad inventory on social networks, is barely breaking even despite selling 3 *billion* ads per month.  And rather than raising prices to become profitable, they're actually in the uncomfortable position of lowering prices 40% -- from 12.5 cents to 7.5 CPM.  It reminds me of the (often unintentional) joke "We lose money on every transaction, but we make it up in volume!"

All this has made them so gloomy about the prospects of their core business that they're thinking of switching horses mid-stream and resurrecting that Web 1.0 favorite: selling demographic data.  I mean, it worked so well the first time, why won't it work now?

Ok, so you might be saying "Sure, social network ads are crap, but Google's ads are solid, right?  After all, they're set by the open market!"  I thought that too, until recently I learned that rather than that market being open, it in fact is restricted by a series of minimum bids. 

Don't believe me?  Search for "Flash" and you'll see it has zero ads.  In a totally free market, that means you have no competition, and thus should be able to bid as low as you want to get your ad to appear.  But when you try to create an AdWord for the "Flash" keyword, you'll see it sets the minimum price at $0.10.   So even if the market (me) only wants to pay $0.01, it's priced 10x higher than the market (I) will bear.  Which is why there are no ads on the "Flash" keyword.

Said another way, there is no competitive pricing for the "Flash" keyword.  Rather, the price is arbitrarily set by Google.

Now you might say "Well, Google owns the ad inventory, they can sell it for whatever they want; it's still a free market, even if they choose not to sell it for cheap."  But wait -- didn't we just say prices are set by auction?  Hm...

All this means that the auction only sets prices above a minimum.  Which brings us to the $149.86B question: how many of Google's ads have prices set by auction, and how many are just coasting by on the minimum?

Hopefully, most of Google's ads are competitively priced via the auction.  This would suggest that they're priced "correctly" and that we're in for no major shocks to ad revenue (and, due to Google's market share, worldwide ad revenue).

But let's say that some high fraction -- 50%?  70%? -- of Google's ads are in fact not competitively priced, but are just set arbitrarily by Google, such as Flash's $0.10 minimum.  In this scenario, Google's revenue is no more protected from price declines than Lookery and it's 40% "going out of business" sale.  (After all, anybody slashing prices while losing money can't have a long future.)

In this scenario -- which might be reality, depending on the data -- Google's pricing is not sustained by competition, but by a near monopolistic control of ad inventory.  And in that maybe-reality scenario, the global ad market is over-priced, meaning Google and all other ad-supported online businesses are overpriced, meaning we're in for another massive internet economy collapse if Google ever loses its monopoly and is forced to truly compete on price.

Sound crazy?  It's not nearly as crazy as what's already happening in reality: ad arbitrage.  It works like this:

You buy a really cheap adword from Google in order to direct a lot of traffic to some site.  And then you fill that site with ads with high CPM and CPC (perhaps from other ad networks, it doesn't matter).  The result is you buy a click for $0.10, and then turn around and sell it for $1.00.  How is that possible?  Why isn't everybody doing it?

Everybody was doing it -- in huge quantities -- until Google killed it.  How?  By raising minimum bids.

That's right, by fiat Google leveraged its near monopoly power and raised prices to stop buyers it didn't like (spammers) from taking advantage of the mysterious imbalance between the price of an AdWord and another network.  Specifically, it means that Google clicks should be able to buy for *cheap*, were it not for Google artificially raising prices.

It also means that advertisers on other networks are unwittingly paying *way too much* for what should actually be a really cheap click.  In an efficient market, the advertiser should just be buying that original, really-cheap AdWord, rather than the inflated price of an ad on the intermediate spam site.

And all this comes back to the possibility that Google's AdWord inventory is actually overpriced, and it's only sustainable so long that Google enjoys near-monopoly status.  Once that status is gone, then all keywords -- even the ones Google chooses to price out of the market -- become competitively priced, at rates far lower than what Google is currently charging.  Which means everybody that depends on AdWord revenue suddenly makes less.  Meaning the internet economy collapses.  Again.

Crazy ramblings?  I wonder.


Update: A friend (the same one who told me about ad arbitrage) pointed out that one reason there might be no ads on Flash is because it's a trademark. That could be -- I didn't go all the way through to pay the minimum bid and see what happens. But that doesn't change the fact that there are minimum bids on *all* unused keywords, including such words as "blah" or "quinthar". So the general argument still stands.

Update: So I decided to run my own Flash ad. I gave it a $25/day budget and bid the $0.10 minimum. It immediately showed up, and appeared every time I refreshed for at least several hours. According to Google, it was shown 6,607 times. But here's the interesting thing (which, frankly, completely demolishes my whole theory): it was pulled, despite it only costing me $0.40 (ie, with tons of budget remaining). Why? Because clickthrough was too low -- 0.06%, to be precise. So now I'm changing my theory. The reason there are so few Flash ads isn't that Google has priced the keyword out of the market. Rather, it's because it's difficult to make an ad that achieves sufficient clickthrough on such a general term as Flash. Even if you're willing to pay the minimum, Google isn't willing to show it unless it performs. Whether or not that's a problem (and I'm not sure it is), it's entirely different than what I initially was guessing, and completely undermines my theory of Google using monopolistic pressure to sustain noncompetitive pricing. So... never mind.

I'm not normally one to defend the music industry, but I don't know if I agree with this Techdirt article lamenting how Lyle Lovett sold 4.6 million albums but has "never made a dime" from album sales.  Specifically, I take issue with this line:

"Of course, the truth is that it's quite rare for any musician to make money from selling their albums, as has been pointed out for years"

While that might technically be true, it ignores that Lyle Lovett almost certainly was paid a healthy advance -- precisely due to the anticipation of future sales. 

And lo and behold, the sales rolled in.  It says he's sold 4.6M albums.  That's over a long period where I'm sure prices have fluctuated, but let's say a $10 average album price, so $46M in revenue.  Thus if he had a $10M advance, that means he's earned 21% of revenue from his albums since 1991.

Is that good?  Bad?  Frankly, that sounds pretty good to me.  If he were the founder of a startup, that's like selling for $46M and keeping a 21% stake -- it'd be widely seen as a success.  Granted, that'd be a 17 year startup but still.  Hard to criticize his deal.  I sorta agree with Kurt: even if he's not paid per album, he's still well paid, and especially given he didn't take on any of the risk.

Then again, if his advance were much smaller -- say, $1M, then he's only looking at 2.1% of the album revenue, which isn't very cool at all.

Furthermore, that's only counting albums since 1991.  He had three albums before that that aren't included in the above analysis...

So, it comes down to the numbers, but I don't think it's clear whether he's being screwed or not.  Not to mention, he's still paying down his advance, and one day will probably start making money.  Or maybe not.  But that doesn't change the fact he was (probably) paid a crapload of money up front for a service that had yet to be performed.  We should all be so lucky to be in such an unfortunate position.

Any idea what's up with this new plan from ThePirateBay to "encrypt the internet"? Is this trying to replace the TCP stack with an SSL stack -- every time you try to open a TCP connection it instead first tries to (essentially) open an SSL connection and, failing that, falls back onto regular TCP?

Amazingly, Wikipedia doesn't even have a page for it -- crazy!

This from the IPTEE website:

"As an example, when a new TCP connection is established with the 3-way handshake the crypto layer at the "initiator" will hold any pending application data and start the key negotiation process. When a session key has been established, the pending application data is encrypted and sent. All following communication on that TCP stream is then treated as encrypted traffic. Should the remote host not reply with a valid key-negotiation response, the TCP connection is closed, re-opened, and the application data passed through unencrypted."

Ug, so when this rolls out, every single TCP connection will start with a garbage request, fail, and then reconnect? This seems like it could lead to an *exceedingly* slow experience for anybody who installs this, as 99.999% of all TCP connections they attempt will fail, and probably not quickly but rather by timeout.

For example, the servers I write typically accept a TCP connection, and then keep receiving as much as possible until it gets a well-formed request. If it doesn't get a well formed request it'll eventually timeout. But -- and this is key -- a malformed request won't be instantly detected and trigger a disconnect. Indeed, this would be extremely difficult in the general case.

For example, assume your protocol used an HTTP-like structure with \r\n terminated lines, the first of which is just a big string, and each subsequent line is a "Header: value" pair, followed by a blank line (double \r\n). Super basic protocol, like this:

GET / HTTP/1.1
Host: quinthar.com

However, because the parser is general purpose, it doesn't know every possible valid "method" line. The parser takes anything that fits the pattern and passes it to a higher level for processing. This will therefore accept anything like:

POST / HTTP/1.1
Host: quinthar.com

Or even:

The quick brown fox jumped over the
Lazy: log

Normally that's fine -- after all, it's a rare situation for somebody to connect to you with a malformed protocol in the first place, and generally they're going to at least make a good effort so even if it's not totally perfect, leave it to a higher level to figure it out:

Get / Http/1.1
HOST: quinthar.com

But any user with this thing running will result in a hung connection. Rather, it's going to send some binary garbage like this:

sj459dofsfj2q9fdjqre

The parser will look at this and say "ok, it looks wierd, but maybe it's good" and it'll wait until it either gets a well-formed message, or its input buffer is exceeded, or a timeout occurs. In the case of IPETEE, it'll probably always be the last one. Meaning every TCP connection opened to my server will first have this incredible timeout before it gives up and falls back to TCP.


Ok, so now you're thinking "so what David, your servers suck anyway" That may be, but I note that Apache and IIS behave the same way. This means that virtually every HTTP request you make is going to be incredibly delayed. At least, until everybody in the world adopts this.

And that's just HTTP. I'm guessing this design pattern of protocol parsing is exceedingly common. Which means basically all TCP connections of all kinds would be incredibly delayed.

And all this ignores, of course, that this encryption layer doesn't even attempt to prevent man-in-the-middle attacks, so it only provides protection against passive observation. But any ISP that is seriously curious about what's going on inside could simply intercept each end and insert a sniffter. Let's say they do this in 0.001% of all TCP connections, over time they'll figure out who the pirates are (because BitTorrent opens up so many damn TCP connections, they'll come up pretty quick).

The upshot is I'm not sure that this is really a good idea. Anybody who installs it will be horribly slowed. Anybody who runs a server will see a crapload of timeouts and dead requests. And any ISP that really cares about observing users can still easily do so.

Encryption is a really hard problem. The only thing worse than no encryption is encryption that doesn't work, as it leaves you worse off than when you started.

-david

A new article in the Washington Post entitled "Justice Breyer Is Among Victims in Data Breach Caused by File Sharing" talks about how some idiot accidentally shared 2000 social security numbers of a lawfirm's high-profile clients. The article irks me for a couple reasons.

First, the leak is really quite small and insignificant, but the article blows it up like it's a huge thing. Sharing 2000 social security numbers of rich dudes is bad. But it's nothing compared to thousnds of hacked ATMs stealing card numbers *with PINs*, and sending them to a Russian hacker who has been draining bank accounts and has stolen at least $5 million *so far*, and hasn't yet been stopped. A little context please? (And the context provided in the article comparing it to a few other insignificant leaks isn't exactly helpful.)

But what bothers me even more is this completely false statement:

Robert Boback, chief executive of Tiversa, the company hired by Wagner to help contain the data breach, said such breaches are hardly rare. About 40 to 60 percent of all data leaks take place outside of a company's secured network, usually as a result of employees or contractors installing file-sharing software on company computers.

First, I don't even know what that means: how can it both be "outside a company's secured network" and "on company computers"? Or does "secured network" mean "the subset of the network that happens to not leak yet"? (Or does "network" mean "the office internet connection", without including the computers that connect to it?)

Regardless, it claims 40-60% of "all data leaks" are "usually as a result of ... file-sharing software". Where does that data come from? The only really exhaustive study I know on the subject was the Verizon one, and it came to a completely different conclusion:

Specifically, the words "p2p" and "file-sharing" and "limewire" don't appear anywhere in it. Furthermore, it says only 18% of leaks are due to insiders, and of those, only 3% were "inadvertent disclosure" (which I think would include accidentally sharing something on Limewire).

The upshot is the Verizon study suggests the exact opposite as this article: rather accidental file sharing being a significant source of leakage, it accounts for at maybe 0.54% of leaks.

So... what's up with the anti-P2P FUD?

I've been a long qemu fan, and a new debootstrap groupie. The following script was built with the help of the debian-user list and combines these two affections -- with some fancy footwork -- to build a "from scratch" bootable qemu image. There are still some kinks to be worked out (specifically, I don't know where to get a "real" stage1, stage2, and e2fs_stage1_5 file, so it copies it from the host system), and right now it is fixed at 1GB (though that could be easily corrected), but otherwise it seems to work surprisingly well. Behold:

echo "Creating 1GB file of zeros in $1.raw"
dd if=/dev/zero of=$1.raw bs=1024 count=1048576

echo "Formating $1.raw with ext2 filesystem"
/sbin/parted $1.raw mklabel msdos
/sbin/parted $1.raw mkpart primary ext2 0 954
/sbin/parted $1.raw mkpart extended 954 1069
/sbin/parted $1.raw mkpart logical linux-swap 954 1069
/sbin/parted $1.raw set 1 boot on
/sbin/parted $1.raw mkfs 1 ext2

echo "Mounting $1.raw on $1.mount"
mkdir -p $1.mount
sudo mount -o loop,offset=16384 -t ext2 $1.raw $1.mount

echo "Installing Etch into $1.mount"
sudo debootstrap --arch i386 etch $1.mount http://ftp.us.debian.org/debian

echo "Setting up host networking in $1.mount for apt"
sudo cp /etc/resolv.conf $1.mount/etc
sudo cp /etc/hosts $1.mount/etc

echo "Installing kernel into $1.mount"
sudo chroot $1.mount apt-get update
sudo chroot $1.mount apt-get -y install gnupg
sudo chroot $1.mount apt-get update
echo "do_symlinks = yes
relative_links = yes
do_bootloader = yes
do_bootfloppy = no
do_initrd = yes
link_in_boot = no" > /tmp/kernel-img.conf
sudo mv /tmp/kernel-img.conf $1.mount/etc
sudo chroot $1.mount apt-get -y install linux-image-2.6-686

echo "Manually installing grub into $1.mount"
sudo mkdir -p $1.mount/boot/grub
sudo cp /boot/grub/stage1 $1.mount/boot/grub
sudo cp /boot/grub/stage2 $1.mount/boot/grub
sudo cp /boot/grub/e2fs_stage1_5 $1.mount/boot/grub
echo "default 0
timeout 0
title Linux
root (hd0,0)
kernel /boot/vmlinuz-2.6.18-6-686 root=/dev/hda1 ro
initrd /boot/initrd.img-2.6.18-6-686" > /tmp/menu.lst
sudo mv /tmp/menu.lst $1.mount/boot/grub
sudo echo "device (hd0) $1.raw
root (hd0,0)
setup (hd0)
quit" > /tmp/grub.input
sudo grub --device-map=/dev/null < /tmp/grub.input

echo "Configuring qemu networking"
echo "auto lo
iface lo inet loopback
allow-hotplug eth0
iface eth0 inet dhcp
" > /tmp/interfaces
sudo mv /tmp/interfaces $1.mount/etc/network

echo "Starting sshd and granting $USER a root key"
sudo chroot $1.mount apt-get -y install ssh
sudo chroot $1.mount /etc/init.d/ssh stop
sudo mkdir -p $1.mount/root/.ssh
sudo cp ~/.ssh/id_rsa.pub $1.mount/root/.ssh/authorized_keys
sudo chmod -R 755 $1.mount/root/.ssh

echo "Dismounting $1.mount"
sudo umount $1.mount

echo "Done. To start, run:"
echo ""
echo " sudo qemu -kernel-kqemu -redir tcp:2222::22 $1.raw"
echo ""
echo "To SSH in, run:"
echo ""
echo " ssh -p 2222 root@localhost"
echo ""

I hadn't heard of this implementation before -- or the term "brightnet" -- but the general idea is to split up files into randomly XOR'd chunks, and then share those chunks via P2P.  (I think some of the blocks are also pure random data, to confuse things further.)  This means a single collection of blocks can generate multiple files, and there's no way to know from the outside which you want.

The technique thwarts the BitTorrent attack where you know someone has the file by the mere fact that you can download it from them.  In this model, merely being able to download a block from a peer doesn't mean the peer has all the other blocks of that file, nor does it mean that that peer is using the block for illegal purposes. The same exact block could be used to legally construct a public domain song, or illegally construct an unlicensed copy of a copyrighted song.

Anyway, just more proof (as if we needed it) that the end of copyright is near.  It'll still exist on the books, but just become increasingly unenforceable -- one of those quaint anachronisms that we'll scoff at while reminiscing with our grandchildren.

-david

PS: I alluded to this concept in a post to a mailing list on 6/26, but I had no idea any implementation was so advanced.  My original post follows, and is in response to a poster claiming that even without the "making available" argument, the RIAA has plenty of tools to wage its anti-pirate campaign:

This seems to depend on three things:

1) Licensed copies can be distinguished from unlicensed copies.

2) It's possible to know who you downloaded a given file from.

3) Running a P2P service is generally regarded as shady activity

All these might be generally true now, but I think the trends work against all three.  Once an enforcement regime that depends on any or all of the above comes into force, the pirates will just switch systems.

Granted, I think you're right: with enough work and forensic analysis and circumstantial evidence you'll be able to prove it to a jury.  But it'll get really expensive to do this -- especially because pirate systems no longer advertise everything you've ever downloaded, and thus it's impossible to distinguish between a one-time and hard-core pirate (without just downloading an incredible amount of pirated material and looking for repeat offenders -- though with changing IP addresses and no permanent identifier, that gets hard).

The upshot is it might be a rather Pyrrhic strategy where the cost of suing a group of people exceeds the damages you get from the subset of people you win against.

----

Though not super related, it's a fun exercise to think how to develop a system that evades the above 3 forensic trails.  I'd toss out:

1) Converge pirate networks on perfect duplicates of legitimate copies that are available somewhere online.  Even if there are commercials embedded, come up with "metadata" that notes where the commercials are and program players to automatically skip over them.  Create MP3 ripping tools that explicitly create binary identical files even when ripped by different people, thereby enabling the argument that you ripped it and threw the CD away.

2) Use onionskin routing to obscure the trail to the actual host of the content.  Use file sharding such that everybody hosts a tiny fraction but nobody hosts the whole thing.  XOR file shards such that the only way you can get a particular file shard is to combine two entirely different ones, so nobody is hosting even subsets of the file directly.

3) Build a P2P system that has both legitimate and illegitimate purposes.  Have it implicitly "share" your entire hard drive, but it only actually responds to files with a given hash (thus any private information is implicitly protected because nobody knows its hash).


Taken all together, (1) makes your pirated content look potentially legitimate, (2) hides you when others download from you, and (3) lets you argue you're not a pirate but just enjoying a legitimate P2P network and -- golly, you didn't know it could be use for piracy!  None of these are rock-solid defenses, but it's not really protecting against a rock-solid attack, either.  The RIAA campaigns today are at best a break-even endeavor (when all the destruction of public sentiment is weighed in as a cost) -- if the cost could be magnified 2x, 5x, or 10x, then even they will give up.

-david

If the "making available" argument fails, does anyone feel there is any
realistic way for a copyright holder to prove infringement?

Any modern pirate system explicitly eliminates paper trails -- soon
they'll all use encryption and, if that's not enough, onionskin routing.
(Bandwidth is growing way faster than the size of MP3; eventually
onionskin will be perceived as cheap.) And if that doesn't work, I have
no doubt something else will be invented.

So it's safe to assume that any pirated tools in the future will, for
all intents and purposes, cause music and movies to just magically
appear on demand from an unknown, unprovable source.

In this world, even if widespread piracy is obviously occurring, would
there be any way to prove it without "making available"?

I'll take a stab at my own question and say "yes", but the shift will go
from pursuing distributors to pursuing downloaders. And I think they'll
next try some sort of "proof of payment" scheme, such as used by public
transportation:

In San Francisco, there are MUNI trains that you can board anywhere and
get off anywhere; there's no physical requirement to buy a ticket.
However, you're legally obligated to have one, and if you fail a spot
inspection by an officer of the law, you'll pay serious fine.

I wonder if that's the model they will attempt next if "making
available" fails. Basically, all stores will move to
individually-tagged songs and movies where proof of purchase is encoded
in the content itself. (This is impractical in the old world of
physical media distribution, but becomes more feasible as we move to
on-demand downloads).

One way to do this would be with watermarks: so long nobody has
incentive to remove them, they'll stick around fine. But then again,
you could probably do it with just ID3 tags and digital signatures (a
message "Bob has bought track <SHA1>" signed by Time Warner's public key
would suffice). Technically it's an easy problem to solve.

The problem will come in the audit: both how to audit the devices in
question, and when to do it.

As for how, the challenge (as always) is to distinguish between content
in the public domain and content you need permission from the copyright
owner to have. One possibility would be to build an opt-in waveform
fingerprint of all copyrighted works that elect to participate in this
proof of payment scheme. This won't truly catch everything (and won't
catch anything released before the scheme launched), but even if it
catches only the new releases with some regularity, that starts to make
an effective tool for general compliance enforcement.

So, auditors could conceivably have a device that has USB and iPod
connectors that plug into basically anything, scan all content for
waveform matches, confirms the file has a proof of payment certificate,
and alerts if not.

Ok, so all this could technically be built by a sufficiently incented
(or incensed?) party. This brings us to the next question: when would
the audit occur?

This is where it'd probably fail on constitutional grounds. A scan
under most circumstances would be "unreasonable search and seizure".
But one place that is notoriously exempt: border control. They can
basically take anything and do anything for as long as it takes.

Granted, this cedes the vast majority of domestic piracy. But their
goal isn't to eliminate the potential for piracy; their goal is to make
it such a pain that people still choose to buy. If they first make it
impossible to travel internationally without first cleansing all devices
of pirated works, this will start to bite. And after that, they'll find
other excuses to audit devices: airport security for domestic flights?
PCI and SOX compliance audits? Build auditing straight into the iPhone
itself?

The big question in my mind is whether everybody just gives up on
copyright before then and "just says no" to proof of payment and spot
copyright checks.

By and large, society as a whole has already given up on copyright, as
evidenced by overwhelming adoption of piracy. It's possible that if
pressed to make a decision that we'll simply refuse to pass any law that
allows for reasonable enforcement. Then businesses that depend on
enforcement will die and get replaced with those that don't, and
gradually the courts will limit the scope of copyright to where it can
be realistically enforced.

Anyway, so I see a copyright-free (or copyright-very-limited) future as
a legitimate possibility. And society might just refuse to allow the
proof-of-payment scheme to go into force.

So, let me conclude with my prediction: if "making available" fails (and
if they truly accept this -- not necessarily a sure bet), then major
copyright holders will marshal their forces and attempt to create a
"proof of payment" system with enforcement starting at border crossings
and gradually increasing from there. This will trigger a showdown with
society at large as it really begins to weigh how much it cares about
copyrights, and the people who hold them. And I think it's very
possible that society decides the cost of copyright enforcement
outweighs its benefit and essentially curtail copyright in all areas
where it stopped making sense, long ago.

-david

With all that asynchronicity, it behaves in... unnatural ways. Consider the following:

var message = "Hello";
var blah0 = function() { alert( message ); }

message = "World";
var blah1 = function() { alert( message ); }

blah0();
blah1();

In any reasonable language, it would first alert "Hello", and then alert "World". But noooo. Javascript's gotta get all fancy and defer execution until later. And at that point message is equal to "World" for both. So it's "World" "World", which ain't right at all.

I still don't know how to do the above, but thankfully Tyler turned me on to a new path with "this":

var div0 = document.getElementById("div0");
var values = [ "Hello", "World" ];
for( var c in values )
{
var input = document.createElement("input");
input.type = "checkbox";
input.value = values[c];
input.onclick = function() { alert( this.value ); }
div0.appendChild( input );
div0.appendChild( document.createTextNode( values[c] ) );
div0.appendChild( document.createElement( "br" ) );
}

Frickin' Javascript...

I'm fascinated by the notion of software defined radios.  In particular, I'm fascinated by the potential for hackers to take over the radio waves like they have so effectively dominated the internet.  Consider the following futurological daydream:

1) FCC mandates all US cellphone carriers must accept any compatible device onto their networks.

2) TuxPhone quickly establishes connectivity with the major carriers.

3) TuxPhone adopts support for the Android application stack and becomes a feasible (albeit chunky) daily cellphone for hackers.

4) TuxPhone is ported to the GNU Radio platform, thereby replacing "pre-approved hardware modules" with firmware/software.

So the above isn't too far fetched -- all those guys have an interest in joining forces, and everything they're doing is in line with conceivable law (assuming (1) happens).

But once the above is done, there will be an easily available platform for hackers to ply their trade.  In particular, there will be a widely available, low-cost platform with sufficient "non-infringing purpose" such that the devices themselves cannot be restricted.  At first it'll start out relatively benign:

5) .public adopts TuxPhone as the development platform of choice for real-world testing of a completely encrypted, wireless, decentralized communication mesh.

But then it'll start to get more interesting:

6) A researcher submits a patch to .public enabling frequency-hopping spread spectrum communication, clearly marked in such a fashion that it would be illegal to use it outside of certain, default frequency ranges.  But those ranges will be user definable, and it will be easy to override the default and set the range to "all".

7) Someone packages it into an extremely simple application where anybody can enter in a password that serves as the "seed" for the pseudorandom frequency generator.  Any two individuals (in radio range) with the same password can thus generate the same random set of frequencies and thereby hop in sequence to the same frequencies.  (Though realistically it'll first just establish an SSH session using public keys over a standard narrowband connection, exchange the symmetric key (aka random seed), and then jump.  Passwords are so passe.)

8) Somebody figures out how to make a spread-spectrum repeater that also happens to serve as a fantastic narrowband repeater.  Thus mid-range mesh communications gain the benefits of pseudorandom frequency hopping, but in a way that isn't obviously illegal (because it also enables legal narrowband meshes).

9) As technology improves, there becomes no reason *not* to broadcast at the maximum legal power.  Thus the coverage and capacity of each node in the mesh increases.  In parallel, as costs fall and average utility increases, the density of the mesh also increases.  Taken together, identifying the precise location and identity of any given node becomes more and more difficult -- encouraging more egregiously illegal behaviors as enforcement fails.

10) Ultimately, mesh coverage is "good enough" for anybody to exchange high-bandwidth information with anybody within a hundred miles (especially in urban areas, where most people live).  On top of this, secure tunnels through "global" networks (satellite, fiberoptic) become as plentiful as anonymizing proxies are today.


Witness the start of perfectly secure, untraceable, global communication.


Legally, this starts to blur the meaning of spectrum ranges, which the major carriers have paid billions to lock up.  But technically, it'll be very hard to stop.  I'm curious if the futile battle for spectrum rights will play out in a similar way to the currently futile battle for copyright.

All this despite the fact that spread spectrum technology *obviates* the need for fixed spectrum bands.  The resulting technology might actually be *better*: more reliable, higher capacity, cheaper, and more flexible than narrowband technology.  Thus while the "old guard" becomes increasingly dependent upon legal war to maintain their narrowband business models (tied to ridiculously expensive spectrum licenses), the hackers will have shown that they're no longer needed -- just like they're doing today with copyright.

But the most unfortunate re-enactment of all will be if the full potential for spread-spectrum, frequency-hopping, software-controlled radios is stymied in its most crucial phases by entrenched interests, preventing any legitimate business investment in this groundbreaking technology and instead driving it underground into the hands of pirates and insurgents.

Sound familiar?

The debate over network neutrality is just a rehash of the age-old debate between top-down and bottom-up design.  Bottom-up invariably wins.  And it will here too.

The technical critics of network neutrality typically emphasize how it comes at a cost in theoretical efficiency.  The same was said in defense of communism.  And waterfall design.  All were wrong.

Or, rather, they were right in the land of theory, where code compiles on first try and rivers run with pure gold.  In this magical land of brilliant architects and benevolent dictators, top-down planning works.

But in our world, the risk of error, mis-management, and outright abuse is way, way too high.  The theoretical capabilities of top-down design are -- more often than not -- lost to the practical realities of waste, incompetence, and corruption.

Thus the whole meta-debate obscures the broader point: we shouldn't be designing for maximum efficiency.  We should be designing for minimal corruptability.  Once that's in place, we can figure out the rest.  But if that's not in place, then nothing else matters because even the best design will eventually be corrupted.

So back to the topic at hand: network neutrality does forbid certain types of filtering that could in theory improve certain types of applications.  In particular, VoIP and HDTV streaming are always trumped out.  Without network neutrality, we cannot perfect these systems -- the latency and jitter are just too high!

But this technical point is not only wrong (Skype already does VoIP, and Akamai already does HDTV streaming), it mistakenly (or disingenuously?) ignores the far more serious potential for corruption that such filtering enables.

It's always a treat to play with JavaScript. C++ will always have my heart, but I do enjoy the an occasional tryst with JavaScript. I think what's best about JavaScript is the HTML DOM. It... it feels so dirty using it, just making raw manipulations on the structure of the web itself. It seems there should be some protection involved -- surely you can't just blast over half the page with innerHTML? But you *can*. That's what's so amazing about it. No matter how much abuse you do to it, it just keeps on going. Null-pointer dereference? It's cool. Syntax error? Meh. It just keeps going.

That's not to say that "it just works", however. Going and working are two different things. But it does get surprisingly close.

Did you know that the ratio of pirated music to legitimately downloaded music is roughly akin to the ratio of salt water to fresh water on the planet?  I noticed this in a moment of exasperation when reading a post casually suggesting how piracy should be reduced to a "livable range".  When translated with the above observation, the statement is about as realistic as:

"We all have to live with a range of salinities... I think the goal is to relegate salt-water to a livable range not to eradicate it altogether, which is likely impossible anyway."

Piracy *is* the digital music industry.  Everything else is a footnote or a rounding error.

I recently heard a statement to the effect of "total dollars spent on advertising probably won't go down", and I wonder about that: though both on and offline ad revenue is growing, online grew 7x faster than offline (28% vs 4%).  Furthermore, over 90% of ad spending is still offline.  So global ad growth is really quite modest -- despite internet's impressive numbers.

The upshot is the vast majority of ad dollars have yet to make the switch.  But if advertising gets more precise online, then the cost to achieve a certain effectiveness of advertising should go down.

The result is the total dollars required to achieve the same level of global ad effectiveness might be substantially less than is currently being spent.  So even if the online growth continues to skyrocket, it might precipitate an equally or even more dramatic decline in offline ad spending.

Basically, if the improved effectiveness of online ads exceeds the global growth of the ad market, then total dollars spent on advertising will go down.  And frankly, I'd wager online ads are substantially more effective than offline -- by 20%?  50%?  If so that means total ad spending might have a long way to fall.

http://www.nytimes.com/2008/06/08/arts/music/08pare.html?_r=1&scp=1&sq=trent&st=nyt&oref=slogin

Pretty long article, but two parts jumped out for me:

> Mr. Reznor has no global solution for how to sustain a long-term
> career as a recording musician, much less start one, when listeners
> take free digital music for granted. "It's all out there," he added.
> "I don't agree that it should be free, but it is free, and you can
> either accept it or you can put your head in the sand."

I like how Trent appears to reluctantly accept the new constraints of
reality, and then work within them. Personally I think the new
"constraints" are actually more liberating than the past ones, but I
don't think it's necessary to believe this: rather, it's only necessary
to believe that the constraints have changed, whether you like it or not.

> When record labels didn't want it, Mr. Reznor put it online: free to
> the first 100,000 downloaders as good-quality MP3 files or $5 for
> more high-fidelity files. He had thought that fans would willingly
> pay the price of a latte to support musicians directly. But fewer
> than 20 percent did so. "I think I was just naïve."
>
> At the time he called the project a failure, but he has reconsidered.
> "The numbers of the people that paid for that record, versus the
> people that paid for his last record, were greater," he said. "He
> made infinitely more money from that record than he did from his
> other one. It increased his name value probably tenfold. At the end
> of the day, counting free downloads, it was probably five or six or
> seven times higher than the amount sold on his last record. I don't
> know how you could look at that as a failure."

Similarly, I like how he's actively rethinking what success means in
this world of new constraints, rather than just trying to impose his
previous expectations.

All in all, it really sounds like he gets it: not as an wide-eyed
evangelist, but as a down-to-earth realist.

-david