1. Home
  2. Embedded Programming
  3. Embedded BitTorrent: 4 of 5

Embedded BitTorrent: 4 of 5

Embedded BitTorrent: 4 of 5

Inside the Attack that Crippled Revision3

29 May 2008 by Jim Louderback, CEO of Revision3
formatting link

As many of you know, Revision3's servers were brought down over the Memorial Day weekend by a denial of service attack. It's an all too common occurrence these days. But this one wasn't your normal cybercrime - there's a chilling twist at the end. Here's what happened, and why we're even more concerned today, after it's over, than we were on Saturday when it started.

It all started with just a simple "hi". Now "hi" can be the sweetest word in the world, breathlessly whispered into your ear by a long-lost lover, or squealed out by your bouncy toddler at the end of the day. But taken to excess - like by a cranky 3-year old-it gets downright annoying. Now imagine a room full of hyperactive toddlers, hot off of a three hour Juicy-Juice bender, incessantly shrieking "hi" over and over again, and you begin to understand what our poor servers went through this past weekend.

On the Internet, computers say hi with a special type of packet, called "SYN". A conversation between devices typically requires just one short SYN packet exchange, before moving on to larger messages containing real data. And most of the traffic cops on the Internet - routers, firewalls and load balancers - are designed to mostly handle those larger messages. So a flood of SYN packets, just like a room full of hyperactive screaming toddlers, can cause all sorts of problems.

For adults, it's typically an inability to cope, followed either by quickly fleeing the room, or orchestrating a massive Teletubbies intervention. Since they lack both legs and a ready supply of plushies, Internet devices usually just shut down.

That's what happened to us. Another device on the Internet flooded one of our servers with an overdose of SYN packets, and it shut down - bringing the rest of Revision3 with it. In webspeak it's called a Denial of Service attack - aka DoS - and it happens when one machine overwhelms another with too many packets, or messages, too quickly. The receiving machine attempts to deal with all that traffic, but in the end just gives up.

In its coverage Tuesday CNet asked the question, "Now who would want to attack Revision3?" Who indeed? So we set out to find out.

Internet attacks leave lots of evidence. In this case it was pretty easy to see exactly what our shadowy attacker was so upset about. It turns out that those zillions of SYN packets were addressed to one particular port, or doorway, on one of our web servers: 20000. Interestingly enough, that's the port we use for our BitTorrent tracking server. It seems that someone was trying to destroy our BitTorrent distribution network.

Let me take a step back and describe how Revision3 uses BitTorrent, aka BT. The BT protocol is a peer to peer scheme for sharing large files like music, programs and video. By harnessing the peer power of many computers, we can easily and cheaply distribute our huge HD-quality video shows for a lot less money. To get started, the person sharing that large file first creates a small file called a "torrent", which contains metadata, along with which server will act as the conductor, coordinating the sharing. That server is called the tracking server, or "tracker". You can read much more about BitTorrent at Wikipedia, if you really want to understand how it works.

Revision3 runs a tracker expressly designed to coordinate the sharing and downloading of our shows. It's a completely legitimate business practice, similar to how ESPN puts out a guide that tells viewers how to tune into its network on DirecTV, Dish, Comcast and Time Warner, or a mall might publish a map of its stores.

But someone, or some company, apparently took offense to Revision3 using BitTorrent to distribute its own slate of shows. Who could that be?

Along with where it's bound, every Internet packet has a return address. Often, particularly in cases like this, it's forged - or spoofed. But interestingly enough, whoever was sending these SYN packets wasn't shy. Far from it: it's as if they wanted us to know who they were.

A bit of address translation, and we'd discovered our nemesis. But instead of some shadowy underground criminal syndicate, the packets were coming from right in our home state of California. In fact, we traced the vast majority of those packets to a public company called ArtistDirect (ARTD.OB). Once we were able to get their Internet provider on the line, they verified that yes, indeed, that Internet address belonged to a subsidiary of ArtistDirect, called MediaDefender.

Now why would MediaDefender be trying to put Revision3 out of business? Heck, we're one of the biggest defenders of media around. So I stopped by their website and found that MediaDefender provides "anti-piracy solutions in the emerging Internet-Piracy-Prevention industry." The company aims to "stop the spread of illegally traded copyrighted material over the Internet and peer-to-peer networks." Hmm. We use the Internet and peer-to-peer networks to accelerate the spread of legally traded materials that we own. That's sort of directly opposite to what Media Defender is supposed to be doing.

Who pays MediaDefender to disrupt peer to peer networks? I don't know who's ponying up today, but in the past their clients have included Sony, Universal Music, and the central industry groups for both music and movies

-- the RIAA and MPAA. According to an article by Ars Technica, the company uses "its array of 2,000 servers and a 9GBps dedicated connection to propagate fake files and launch denial of service attacks against distributors." Another Ars Technica story claims that MediaDefender used a similar denial of service attack to bring down a group critical of its actions.

Hmm. Now this could have been just a huge misunderstanding. Someone could have incorrectly configured a server on Friday, and left it to flood us mercilessly with SYN packets over the long Memorial Day weekend. If so, luckily it was pointed at us, and not, say, at the intensive care unit at Northwest Hospital and Medical Center But Occam's razor leads to an entirely different conclusion.

So I picked up the phone and tried to get in touch with ArtistDirect interim CEO Dimitri Villard. I eventually had a fascinating phone call with both Dimitri Villard and Ben Grodsky, Vice President of Operations at Media Defender.

First, they willingly admitted to abusing Revision3's network, over a period of months, by injecting a broad array of torrents into our tracking server. They were able to do this because we configured the server to track hashes only - to improve performance and stability. That, in turn, opened up a back door which allowed their networking experts to exploit its capabilities for their own personal profit.

Second, and here's where the chain of events come into focus, although not the motive. We'd noticed some unauthorized use of our tracking server, and took steps to de-authorize torrents pointing to non-Revision3 files. That, as it turns out, was exactly the wrong thing to do. MediaDefender's servers, at that point, initiated a flood of SYN packets attempting to reconnect to the files stored on our server. And that torrential cascade of "Hi"s brought down our network.

Grodsky admits that his computers sent those SYN packets to Revision3, but claims that their servers were each only trying to contact us every three hours. Our own logs show upwards of 8,000 packets a second.

"Media Defender did not do anything specific, targeted at Revision3?, claims Grodsky. "We didn't do anything to increase the traffic" - beyond what they'd normally be sending us due to the fact that Revision3 was hosting thousands of MediaDefender torrents improperly injected into our corporate server. His claim: that once we turned off MediaDefender's back-door access to the server, "traffic piled up (to Revision3 from MediaDefender servers because) it didn't get any acknowledgment back."

Putting aside the company's outrageous use of our servers for their own profit, and the large difference between one connection every three hours and 8,000 packets a second, I'm still left to wonder why they didn't just tell us our basement window was unlocked. A quick call or email and we'd have locked it up tighter than a drum.

It's as if McGruff the Crime Dog snuck into our basement, enlisted an army of cellar rats to eat up all of our cheese, and then burned the house down when we finally locked him out - instead of just knocking on the front door to tell us the window was open.

In the end, here's what I know:

  • A torrential flood of SYN packets rained down on Revision3's network over Memorial Day weekend.

  • Those packets - up to 8,000 a second - came primarily from computers controlled by MediaDefender, who is in the business of shutting down illegal torrent sites.

  • Revision3 suffered measurable harm to its business due to that flood of packets, as the attacks on our legitimate and legal Torrent Tracking server spilled over into our entire Internet infrastructure. Thus we were unable to serve videos and advertising through much of the weekend, and into Tuesday - and even our internal email servers were brought down.

  • Denial of service attacks are illegal in the US under 12 different statutes, including the Economic Espionage Act and the Computer Fraud and Abuse Act.

Although I can only guess, here's what I think really happened. Media Defender was abusing one of Revision3's servers for their own purposes - quite without our approval. When we closed off their backdoor access, MediaDefender's servers freaked out, and went into attack mode - much like how a petulant toddler will throw an epic tantrum if you take away an ill-gotten Oreo.

That tantrum threw upwards of 8,000 SYN packets a second at our servers. And that was enough to bring down both our public facing site, our RSS server, and even our internal corporate email - basically the entire Revision3 business. Smashing the cookie jar, as it were, so that no one else could have any Oreos either.

Was it malicious? Intentional? Negligent? Spoofed? I can't say. But what I do know is that the FBI is looking into the matter - and it's far more serious than toddlers squabbling over broken toys and lost cookies.

MediaDefender claims that they have taken steps to ensure this won't happen again. "We've added a policy that will investigate open public trackers to see if they are associated with other companies", promised Grodsky, "and first will make a communication that says, hey are you aware of this."

In the end, I don't think Media Defender deliberately targeted Revision3 specifically. However, the company has a history of using their servers to, as Ars Technica said, "launch denial of service attacks against distributors." They saw us as a "distributor" - even though we were using BitTorrent for legitimate reasons. Once we shut them out, their vast network of servers were automatically programmed to implement a scorched earth policy, and shut us down in turn. The long Memorial Day weekend holiday made it impossible for us to contact either Media Defender or their ISP, which only exacerbated the problem.

All I want, for Revision3, is to get our weekend back - both the countless hours spent by our heroic tech staff attempting to unravel the mess, and the revenue, traffic and entertainment that we didn't deliver.

If it can happen to Revision3, it could happen to your business too. We're simply in the business of delivering entertainment and information - that's not life or death stuff. But what if MediaDefender discovers a tracker inside a hospital, fire department or 911 center? If it happened to us, it could happen to them too. In my opinion, Media Defender practices risky business, and needs to overhaul how it operates. Because in this country, as far as I know, we're still innocent until proven guilty - not drawn, quartered and executed simply because someone thinks you're an outlaw.

-- Jim Louderback, CEO - Revision3

------------------------------------------------------------------------- Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon

Reply to
Guy Macon
Loading thread data ...

Embedded BitTorrent: 5 of 5

Peer-to-peer poisoners: A tour of MediaDefender

18 March 2007 By Nate Anderson
formatting link

A war of attrition

When your company poisons peer-to-peer networks for a living, public relations usually takes a back seat to discretion; quiet is the rule in the P2P content-protection industry. That's why Jonathan Lee, the company's VP of business development, isn't worried that the corporate web site is down when I reach him in his Santa Monica office. "It's kind of ugly anyway," he says.

For a company like MediaDefender, the largest such firm in existence, privacy comes naturally, but a 2005 acquisition by ArtistDirect has encouraged the firm to take its services public as it starts to look beyond its original client base-music labels and movie studios-and dives headfirst into the brave new world of providing legitimate P2P content for advertisers.

Such advertising deals may be the future, but the company's bread and butter continues to be P2P disruption of movies and music downloads. MediaDefender is quite good at this, as it should be after five years of anti-piracy work. Unlike DRM providers that focus on protecting the product, MediaDefender tries to protect the distribution channel-and only for a limited time. Recognizing that it is impossible to shut down the sharing of copyrighted works, the company focuses instead on mitigation. Record labels and movie companies can pay between $5,000 and $15,000 per title for differing levels of protection that extend over different time periods.

For most content owners, MediaDefender's services are needed at the beginning of a product's life cycle. Lee points out that most movies and albums makes the majority of their money in the first few months after release. MediaDefender's value proposition is not that it can stop such files from being shared, but that it can make sharing difficult for a month or two in order to give the legitimate product more traction.

Is it live or is it MediaDefender?

How it works

To work its magic on the various P2P networks, Lee describes four strategies that MediaDefender uses. All four are powered by a back end of

2,000 servers co-located around the world, and the company has contracts for 9GBps of Internet bandwidth. For a 60-person operation, these numbers are (to put it mildly) a bit high, but the scale of its system usually ensures that the company gets prompt attention and good deals when it goes shopping. It also means that employees who stay late after work to game on the corporate LAN always have a good connection.

Those 2,000 servers do four things that MediaDefender refers to as decoying, spoofing, interdiction, and swarming. Here's how they work...

Four main methods

Decoying. This, in a nutshell, is the serving of fake files that are generally empty or contain a trailer. The goal is to make legitimate content a needle in a haystack, so MediaDefender works hard to ensure that its copies of files show up in the top ten spots when certain keywords are searched for. Everything about the file is tailored to look like the work of pirates, from the file size (movies are often compressed enough to fit on a CD) to the naming conventions to the pirate scene tag. With massive bandwidth and plenty of servers, the company has little trouble in getting these decoy files to appear at the top of search results, but decoying has a down side: the bandwidth. Because MediaDefender actually serves these large but bogus files, it incurs a significant bandwidth bill by using this technique.

Spoofing. Spoofing sends searchers down dead ends. MediaDefender coders have written their own software that interacts with the various P2P protocols and sends bogus returns to search requests, usually directing people to nonexistent locations. Because most people only look at the top five search results, MediaDefender tries to frustrate their first attempts to download a file in hopes that they will just give up.

Interdiction. While the first two techniques try to prevent searchers from locating files, interdiction prevents distributors from serving them. The tool is generally used when media is leaked or newly released; the goal is to slow its spread in those crucial first days. MediaDefender servers attempt to create constant connections to the files in question, saturating the provider's upstream bandwidth and preventing anyone else from grabbing the data.

Swarming. Though he acknowledges the BitTorrent networks can be hard to disrupt, Lee points out that MediaDefender can use "swarming" to make life more difficult for users trying to download copyrighted content. BitTorrent works by using a hash file to reassemble a file from many pieces, each of which may have been downloaded from a different user. MediaDefender simply serves up its chunks of these files, but instead of providing the proper data, its chunks contain static or nothing at all. BitTorrent will discard such junk data, but a flood of it can slow a user's download to a crawl.

Does all of this really curtail P2P usage? Lee admits that the company will never stop file-swapping, but says that isn't the point of what it does. Instead, the goal is to make files hard to find for a short period of time so that studios, music labels, and artists can make money from selling the legitimate product. Companies that use MediaDefender's services will often run their own download tests (or contract with one of the firms that does this) to make sure that they are receiving a return on their investment.

Apparently, they are. MediaDefender counts every major music label and most studios among its clients, with the notable exception of Disney. Lee says that initially, his company expected to work largely with trade organizations like the RIAA and the MPAA. When it actually approached them, however, the trade groups were more focused on court cases and Congressional lobbying. While they approve of MediaDefenders's work, the actual contracts are signed directly with labels and studios, many of whom pay millions for the company's services.

A brave new world: advertising

In recent months, MediaDefender has shifted some of its efforts in a new direction: using its P2P technology and massive bandwidth to serve files, rather than stop them. Last year, the company partnered with Jay-Z and Coke in a widely-covered promotion that saw MediaDefender pushing a legitimate piece of Jay-Z concert footage to fans who searched for videos by the artist. In essence, these are "decoys" that contain real content.

The company has also helped promote Vitamin Water commercials that were deemed too "edgy" for network television, along with video game trailers and exclusive P2P remixes. The goal is to diversify-a necessary safeguard in an industry that has few clients. There simply aren't that many major movie studios and music labels, but there are millions of potential clients with fat ad budgets who wouldn't mind reaching the millions of young, tech-savvy people who make use of P2P networks.

Lee says that even music and movie companies have changed their stance in the last few years, and while none condone illegal downloads, they have realized that this is a huge potential market. This is especially true for smaller indie labels, for whom exposure is sometimes more important than legitimate sales. Some of these small firms have actually paid MediaDefender to serve content by their acts, often in response to users searching for a related (but better known) artist.

This mingling of licit and illicit content on P2P networks raises some questions, of course. How are users to know in advance if content is legal or not? Are some labels actually encouraging the use of such networks, even as their trade groups prosecute those who use them? Does serving legitimate content show confusion about what can and cannot be shared and downloaded?

This was, in fact, a major concern that the industry had. For years, content owners refused to place any legal material on P2P networks for fear of legitimizing them. That fear largely vanished in the wake of the Supreme Court's Grokster decision. Once it was well established that such networks could be held liable for copyright infringement, content owners actually felt more free to make use of the networks for legitimate uses of their own.

But anti-piracy work still accounts for 99 percent of MediaDefender's work-work that Lee knows is not popular in all circles. Last year, for instance, the company began recruiting on college campuses for the first time. Students would approach company reps and tell them that they hated what they did. "But five minutes later," says Lee, "they came back and asked us for a job." Hackers, he says, "love screwing with each other," and MediaDefender gives them an impressive platform and some serious bandwidth to hack on. Besides, "you can't get that mad" about what the company does, Lee says with a laugh. "I mean, you're looking for pirated stuff!"

Update:

Various forum posters and bloggers have commented on MediaDefender's "swarming" claim in particular, arguing that BitTorrent's hash-based technology prevents file disruption and that MediaDefender could simply not disrupt the network. We contacted the company for clarification and were told that the details of their BitTorrent work remain secret, but that the company does indeed employ swarming on BitTorrent networks.

Because of the anti-corruption technology on such networks, MediaDefender tries to stall downloads and make files frustrating to grab by serving bad data. The file corruption discussed in the article should not have referenced BitTorrent; such swarming causes corruption only on networks without similar error-checking (the article text has been corrected). MediaDefender's goal with BitTorrent is to slow down transfers. Making them slow enough counts as a "win" for the company, though this does seem like a hollow victory, as the consumer still has the correct file in the end.

------------------------------------------------------------------------- Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon Guy Macon

Reply to
Guy Macon

formatting link

[%X]

Is a reference to this going to appear in RISKS as it would be worthy of a mention there too, especially for those more critical legal applications that may be affected by such practices.

--
********************************************************************
Paul E. Bennett...............
Forth based HIDECS Consultancy
Mob: +44 (0)7811-639972
Tel: +44 (0)1235-811095
Going Forth Safely ..... EBA. www.electric-boat-association.org.uk..
********************************************************************
Reply to
Paul E. Bennett

ElectronDepot website is not affiliated with any of the manufacturers or service providers discussed here. All logos and trade names are the property of their respective owners.