A Protocol for Visiting China (or DEFCON)

The following is my computer protocol for visiting a hostile environment. I actually designed it under the threat model of "What if I needed to visit China", which requires facing two nation-state adversaries (the US and Chinese government) which may have legal access to the computer, but I use it for going to DEFCON.

It may actually be overkill for DEFCON, but as they say "There is kill, and there is no kill, there is no such thing as overkill". I wanted something I knew could work.


The philosophy is twofold, the first is system hardening, while the second is constraining the damage a compromise could do.

I begin with a clean OS install on a newly formatted hard drive. The system is brough fully up-to-date and necessary tools are installed (Firefox, Bro, Click, ipsumdump, Tex, etc) that I will need during my trip. Plastic MacBooks are especially nice, as the hard drive is trivial to change.

I then segregate data. I create a new account on a server I have access to. This account has a new password, and is accessed through a new SSH private key. I create a version control archive on this account which I can also access from my normal account(s), and use this to store the entire working set I will need during the trip, but no more.

Finally, I set up my web browser. I use NoScript, disable flash, disable Java, and tunnel all traffic through SSH. (I use both browser hardening and a tunnel because its easy to screw up and have traffic escape a tunnel, eg, by forgetting to set Firefox to also tunnel DNS through SSH).

This works not because what is present, but what is absent. I do not have access to my mail accounts, normal public keys, or full working set. Not only do I harden my system, but I explicitly limit the working set so that a compromise minimizes the damage. If I need email access during the trip, I will set up a new Gmail account and forward my mail to the new account.

And once I do that, no worries! I may be on a hostile network, but I've taken steps to minimize my vulnerability surface. But I know I'm not perfect, and who knows what zero-days are lurking in my computer. Thus I've limited the potential damage from a compromise: you can't compromise data that doesn't exist.

On Lauren Weinstein...

I was, for a long time, a participant on Lauren Weinstein's "NNSquad" mailing list. There are many important issues in traffic shaping, traffic management, and other related topics. But I had to conclude, reluctantly, that you can't deal with him. His views are those of a zealot, unwilling to compromise, and seems intent on forcing his view of "neutrality".

The two last straws were his censorship policy and his belief that a high usage cap (250GB, what Comcast is doing) is somehow significantly anticompetitive.

His reaction to Comcast's cap mystifies me. Its almost a total victory for his view: Its transparent, its neutral, its not anticompetitive (250GB/month is >7 hours/day of 720p HDTV video delivered over the net.), and because the response is to first warn than terminate customers over the cap, it CAN'T be used in an anticompetitive manner.


And for a list which is supposed to be about "open exchange of ideas", he's an incredibly harsh censor.

This was my "goodbye to the list" message, which he did not print, of course.

He notes that his part of the conversation was for "private consumption". Yet nothing he said in the private discussion is anything different than what he has said publically on multiple occasions.

It is his sandbox. He can say what he wants, and ignore dissenting views. But by the same token, everyone else should be aware that this is how he operates.

To respect any copyright he might possibly claim, I've excluded his sections, replacing them with paraphrases.




Lauren, folks.

I have to conclude the following: This project will be a failure.
Period. Because even if you "succeed", success will lead to
usage-based pricing. You have proven that you will accept nothing
else.

And, Lauren, you really have practiced heavy censorship, not of
personal attacks but of technical discussion.

You have proven unwilling to acknowledge, respond, or publish the
following, which was an on topic, technical discussion of the issues.

How does the following not perfectly mesh with your stated moderation
policies? Yet it seems to have gotten dropped down the memory hole!
You don't want cooperation. You don't want open discussion.

Thus this is "So long": This project will fail, and the mailing list,
due to Lauren's policy of squelching open discussion which doesn't
agree with his preconceived notions, has already failed.



On Tue, Sep 2, 2008 at 9:46 AM, Nick Weaver wrote:
> Replying on-list.
>
> On Sat, Aug 30, 2008 at 6:30 PM, Lauren Weinstein wrote:
>>
>>> Well, Google's low bandwidth, so it doesn't matter.
>>

{Here lauren insists that google is major bandwidth in the aggregate for both user queries and spidering}

> Google's bandwidth in the AGGREGATE is trivial from the end-customer
> perspective compared with the HD video services which you argue about.
>
> Even google apps is light: Start up google docs, start up a new
> spreadsheet. Thats less than 1 MB transfered (there is other activity
> in my mini-tracelet, so 1 MB is an upper bound). And most of that
> should get cached the second time around.
>
> Or 20 seconds of 400 kbps video. Thats IT. Google (sans video) is
> NOT high bandwidth, even when dealing with a lot of customers,
> compared to video applications.
>
> And even from the webserver side, its not THAT bad.
>
>
>>> But cuil is a
>>> counterexample, where it is not only going up against the ISP behavior
>>> that is the threat, but also a huge existing competitor: google!

{Lauren's comment: Cuil is not proven to be viable.}

> Yet you seem to insist that the network uncertanty should prevent it
> from even being funded?!?
>
> "Would Google be successfully funded if it were trying to get off the
> ground in today's Internet environment? A question to ponder."
>
> Answer: YES, because Google competitors, and much higher bandwidth
> services, are both being funded!
>
>>> But lets look at high-bandwidth new ventures, which do directly
>>> compete with ISP offered services. Like, oh, Youtube, Hulu, etc.
>>> Since Hulu wasn't even launched as a joint venture until 2007, well, I
>>> think this is another data point against your hypothesis.

{Lauren calls youtube relatively low bandwidth}

> Hu? You say that "Google is high bandwidth" but "youtube is not"?!?
>
> Yet your claims that 250 GB cap is somehow anticompetitive relies on
> video services which are an order-of-magnitude larger than YouTube in
> bps.
>
> You can't have it both ways. Google, even google apps, are
> lightweights in comparison.
>
> And I use Hulu as a DATA RATE example, and an example of a company
> today which is really stressing the limits of Internet video delivery,
> complete with full Akamization.
>
>>> As for 250 GB, it was not an arbitrary choice, even if it seems so to
>>> you. Rather, its a round number approximation of the 1% heavy-tail
>>> today.

{Lauren asks me where I get this, and asks how I justify Time Warner's proposed 50GB cap}

> On comcast, speculation based on communication with multiple
> individuals in various ISPs, and actually believing comcast's
> statement that <1% would be affected, based on experience with network
> operations. 250 GB is a lot of data.
>
> There is a reason why, even I, as a researcher with an underutilized
> 100 Mbps pipe to the Berkeley campus, with its >1 Gbps pipe out to the
> rest of the Internet, if I had to transfer >250 GB in a research
> project, I'd use Fedex.
>
>
> And have you been listening?
>
> I DON'T justify TW's 50GB cap. That is exactly the cap level you want
> if you want to be anticompetitive: it keeps all the websurfers and
> casuals happy, but it kills any attempt to do a lot of video over the
> net.
>
>
> That you treat the two the same is the heart of your problem: If you
> and your ilk are going to claim that any cap that could be potentilly,
> possably, maby anticompetitive in the future is just as evil as one
> which is anticompetitive today, why should any ISP listen to you?!?
>
> The ISPs are your frenemies, not your enemies. They are delivering an
> incredible service at incredibly low cost. You should work with them.
> Yes, you need to watch them like a hawk, but they are also rational
> actors and can be worked with.
>
> But your reaction to a reasonable cap, by being effectively the same
> as your reaction to unreasonable caps, has made it clear that there is
> no satisfying your position.
>
>
>>> Be thankful the model isn't "Over the limit? Throttle all traffic to
>>> 100 Kbps" instead, because THAT model is far less cost for Comcast, so
>>> there would be an incentive to reduce the threhhold to affect more
>>> users. But in the model of terminate if over limit, if ever more than
>>> a percent or so are affected, Comcast becomes the one with the serious
>>> problem, not Comcast's customers.

{Lauren says he prefers throttling vs cutoff models}

>
> Riddle me this, then. Which is more anticompetitive at preventing
> video over the net:
>
> a) A 250 GB/month cap where, if you go over, the first time you get
> called and the second time you get cut off?
>
> b) A 50 GB/month cap where, if you go over, you get throttled down to
> 400 Kbps?
>
>
> Lets see, the first is high cost to the customer if triggered, but
> also very high cost to the ISP, and only affects a trivial number of
> users today and even tomorrow, assuming 2.5 Mbps 720p video.
>
> The second is low cost to the ISP, mid cost to the customer, but
> pretty much guarentees that video-over-the-net can't be used as a
> significant form of entertainment.
>
> Cutting off users is an activity that a business can only take if they
> really are at levels that are abusive to the network. You WANT the
> reaction to be fully-cutting-off-the user: it greatly increases the
> cost to the ISP.
>
> User-terminating caps are far more neutral than bandwidth-throttling
> caps, because of the cost to the ISP means that a user-terminating cap
> can only be deployed in really extreme cases, especially when one
> considers the multi-service aspects.
>
>
>>> If Comcast stated that "We will increase the threshhold as demand
>>> grows so less than 1% of customers would ever be affected", would THAT
>>> be satisfactory?

{Lauren basically insists he won't believe it}

> What would it take? Auditors? If it was an audited statement, would
> you accept it then?
>
> Because I don't see how you can convince anyone that a cap affecting
> <1% of the users would have a significant anticompetitive effect. It
> EXACTLY meets your criteria below.
>
> Will you accept the following statement:
>
> IF a bandwidth cap affects fewer than 1% of the customers, it is not
> significantly anticompetitive.
>
> Yes or no.
>
>>> Can ANY cap be satisfactory to you?

{Lauren requires that any cap be justified, and complains that we don't have visibility into the networks in question.}

> A lot of the capabilities are directly reversable from the technology.
> Its all Gige with occasional 10 GigE from the DOCSIS hub, its all
> DOCSIS 2 with some 3 rolling out (but the DOCSIS 3 rollout only
> affects downstream, not upstream).
>
> The physics and all are well known, and if you wanted the details of
> just how many customers and how much frequency range is on a user's
> CMTS, look at the DEFCON work on sniffing cable modems.
>
> DOCSIS is a broadcast medium. I suspect that even with encryption
> turned on, you should be able to get all the information you want on
> the actual internals of the cable company's residential networks.
>
>
> Given a user at the 250 GB/month, 8 hr/day duty cycle, that user is at
> 2 Mbps. Since a DOCSIS channel is only ~40 Mbps, that user is tying
> up 5% of an entire cable channel for the whole neighborhood. Thats a
> big cost right there.
>
> Likewise, price out COMMITTED data: price out a T1. Thats a good $100+/Mbps.
>
> Lets assume Comcast's committed rate is 1/5th of that, say $20/Mbps.
> A user at 250 GB is going to use ~1 Mbps continuous, which means at
> MINIMUM, assuming they were at a continual low rate, the user costs
> $20/month. Since in reality users are bursty, AND somewhat diurnally
> synchronized, a 250 GB/month user could easily cost the ISP $60, 80,
> 100+ in transit cost alone.
>
>
> You don't need to trust the ISP's statements to know that a 250
> GB/month user is a severe moneylosing proposition, you just need to do
> a little math.
>
>>> 1) No traffic shaping: best effort only and let the end-points fight it out.

{Lauren says "Voluntary" traffic shaping, well defined, and doesn't skew costs.}

>
> There is no such thing as "voluntary" traffic shaping between users.
>
> And I suspect there really is no satisfying you on traffic shaping either.
>
> EG: a policy like this: "The network enforces fairness such that
> viewed over a time average of X minutes, all users have an equal share
> of bandwidth when congestion occurs".
>
> Now if you talk to Comcast's engineers, and watch their presentations
> at IETF meetings, you'll understand that what they are doing with
> their farness solution is trying to approximate that with simple
> measurement and two QOS bins, so they don't need to buy new equipment.
>
> Yet look at the reaction in this forum to it!
>
>>> 2) flat-rate billing, [1]

{Lauren says there may be cases where usage based pricing is OK}

>
> I am willing to bet that any usage-based pricing scheme that a company
> would deploy would kill your "HD over IP" dreams.
>
> Do you care to take that bet?
>
>>> 3) A significant committed information rate.

{Lauren asks what do I mean by significant}

> You and others seem to subscribe to the "bandwidth is a scarcity"
> arguments, and the "I bought a 16 Mbps download line, I should get a
> good fraction of that", which implies a huge committed information
> rate.
>
> For "significant", you probably mean at least 1 Mbps. Do you want
> your ISP service to cost $100/month more just to give you that?
>
>>> 4) All other services offered by the ISP should be treated as
>>> bandwidth-equivelent with the internet service for 1,2 and 3.

{Lauren notes this is case-by-case}

> If you can't at least approximate these costs with a
> back-of-the-envelope however, you are doing something wrong.
>
>>> [1] And of these, #2 is the greatest threat. Because if you accept
>>> usage-based pricing, that will kill off your future "true HD is 10
>>> Mbps encoding" services faster than you can say "$.20/GB becomes $1/hr
>>> for transport. Have you considered US Mail?"

{Lauren claims UPS is an inappropriate example when comparing data-delivery business models}

>
> It is EXACTLY appropriate, because USPS is the competition for ANY
> "data overnight" video service.
>
> The USPS can get you an incredible amount of data overnight. Lets
> see, a BluRay disk is 50 GB. Thats 4.5 Mbps, and a cost/GB of roughly
> $.02/GB.
>
> If you want "data now", even at just $.20/GB, that is $1/hr for the
> movie, period, with transcoding. Or $10 for a full BluRay disk. Have
> a nice day.
>
> With charges more likely to be on the order of $1/GB, what do you
> think that would do?
>
>
> Remember Tanenbaum's famous maxim: "Never underestimate the bandwidth
> of a station wagon full of mag-tape".
>
>
> This is also why I don't believe in P2P for video content delivery:
> For "data now", it adds bandwidth.
>
> For "data overnight", where it would be friendlier than TCP, then it
> competes with US Mail and US mail's incredibly low cost/bit.
>
>>> [2] And for all the talk about the ISP being an evil monopoly, its
>>> really an evil DUopoly, where the ISP service is often used as a
>>> competitive lever across all services. If there was a huge profit to
>>> still be made in being an ISP, where are the metro-area WISPs? The
>>> third party DSL-based ISPs? They died in the marketplace due to
>>> competitive pressures: there is not much profit margin in being an
>>> ISP.

{Lauren states that the third party ISPs largely died because of the regulatory environment.}

> Start your own. Quit whining and start your own ISP.
>
> The legislative environment has almost no effect on point-to-point
> WISPs. All you need is a tall antenna someplace. A minor headache
> with the local zoning board.
>
> And you can still get DSL lines from the incumbent telco (I do for my
> home service) with layer 3 provided by a third party. That system
> still seems to be working just fine.
>
> I suspect for all the complaints about regulation keeping that duopoly
> intact, the bigger problem is just that the cable/telcos view the ISP
> service as something of a loss-leader: voice and video (either through
> new line or sattelite if you are a telco) is far more profitable, but
> IP service can get people to switch.
>

HTTP is Hazardous to Your Health

The following is not original, but simply a summary of widely known information.

It has been known for decades that plaintext protocols, such as HyperText Transfer Protocol (http) are vulnerable to man-in-the-middle attacks. Yet we are now at the point where there is simply too many ways to man-in-the-middle the web browser, and too much lovely mayhem that can be constructed, for this to be tolerable. We MUST demand that web-sites shift to HTTPS for everything, and ship web browsers that disable http altogether.

How to Man in the Middle: There are simply far too many ways to act as a man in the middle to a web browser. These include maskerading as any access point requested by a system (Karma), ARP cache poisioning (arpiframe), DNS cache poisoning, WiFi packet injection ( airpwn), or simply an ISP attempting to monetize the network (advertisement injection,Phorm). If an adversary can eavesdrop on our HTTP sessions, they can act as a man-in-the-middle.

The problem arises from all the malicious fun that can be done by a man-in-the-middle. This can include:

• Cookie Pillaging: By having the web browser transparently redirect through a long list of sites, the browser will transmit EVERY non-secure (not-SSL-only) cookie to the eavesdropper. Which means the eavesdropper can read! your! gmail!! and other such lovely mayhem, as many sites which allow SSL access don't actually set the cookies properly to mandate SSL access, which means from the viewpoint of an active attacker, SSL does no good at protecting the site!
  
  
• Autocomplete Pillaging: Instead of just redirecting through a long list of sites, include hidden forms and javascript to capture all the autocomplete information present in the browser. A technique developed by H.D. Moore.
  
  
• SMB Reflection: IE will happily open an SMB share when given the proper URL, which can be the attacker's share on the local network. The attacker can use this for the SMB reflection attack (at least on older systems), allowing the attacker on many systems to read and write the user's directory if file sharing is enabled, or to relay authorization credentials to a third party file server. Its unclear how well this can still work, but its at least worth trying.
  
  
• Worms: Take the IE 0-day exploit-du-jour and make a worm that uses packet injection/AP spoofing to spread to all other systems on the local wireless network. For extra credit, release such a worm at JFK airport and include a phone-home visit to the CDC website giving the CDC a nice model for how an influenza-of-doom would spread. (Heathrow may be slightly better, but it is far cheaper to have your worm fly to heathrow in your place. Also, the spread rates in the Usenix paper are probably conservative, because they don't model effects like an infected notebook carrier doing work in a taxi)
  
  
• Drive traffic to your blog:. Gotta have a proof of concept to get people's attention! Note that it only took a couple of hours to hook up a fragile but nontheless working demo. The attack would have been much more effective if I actually played games with wireless transmission power.

So what is to be done? Simple. NO MORE HTTP! Everything, and I mean EVERYTHING should be through HTTPS/SSL. The security community managed to kill off Telnet. We need to do the same to HTTP (oh, and non-secure DNS, too.)

Japanese are going to do "graph takedown"...

Geore Ou reports that the Japanese ISPs are going to start doing something similar to what I noted in january, albeit instead of just attacking the graphs of communication, simply warning and then disconnecting users.

(typo fixed, grr)

A security thought: AT&T Copyright Fighting

The following is just my own opinion and speculation, to a hypothetical question: If I was AT&T, why and how would I implement the AT&T plan to enforce copyright on user traffic. (Note, this post is an extension of my slashdot comment on that thread, and basically describes a "DMCA Takedown on the Network Layer" style of response.)

I also believe this would be a significant problem if implemented. I'm a believer that general network neutrality is a mostly good thing. But when a company seriously proposes filtering, I believe we should attempt to determine what shape such filtering would take, and how it could maximize the stated objectives while minimizing collateral damage. This also gives those opposed to filtering a leg up on attempting to counter it.

To begin with, AT&T probably has a huge incentive to block pirated traffic. Time-Warner cable supposedly has 50% of the bandwidth used by 5% of the users. Who wants to bet that of this bandwidth, it is almost all pirated material and/or pornography? As an ISP, wouldn't you want to remove 1/3rd of your traffic? Especially if its customers that can't really complain about it?

The strength of piracy on the Internet is the ease of getting the pirated material,and the ease of distribution. Thus pirated material must be easy to find if it is to be a substantial portion of traffic and to have a significant economic impact.

So all the MPAA has to do is find the easy-to-find content, and do something about it. Currently, they've tried playing Whak-A-Mole on the Torrent tracking servers, but this has been a losing game, as these servers have already fled to "Countries of convenience", where they are difficult for the MPAA to sue off the network.

But rather than playing Whak-A-Mole on Torrent tracker servers (which are largely offshore), with ISP cooperation from AT&T it becomes possible to play Whak-A-Mole on the torrents themselves. Such a system would benefit both the content owners and the ISPs.

All that is necessary is that the MPAA or their contractor automatically spiders for torrents. When it finds torrents, it connects to each torrent with manipulated clients. The client would first transfer enough content to verify copyright, and then attempt to map the participants in the Torrent.

Now the MPAA has a "map" of the participants, a graph of all clients of a particular stream. Simply send this as an automated message to the ISP saying "This current graph is bad, block it". All the ISP has to do is put in a set of short lived (10 minute) router ACLs which block all pairs that cross its network, killing all traffic for that torrent on the ISP's network. By continuing to spider the Torrent, the MPAA can find new users as they are added and dropped, updating the map to the ISP in near-real-time.

This would be a powerful system, and the likely solution AT&T will use if they carry through on their plans to enforce copyright:

• This requires no wiretapping. Instead, it relies solely on public information: the torrent servers and being able to contact participants in order to map those fetching an individual file. BitTorrent encryption would have no impact on this scheme.
  
• It can be completely automated, both for the MPAA and AT&T
  
• It also minimizes collateral damage, since only participants in an individual torrent can't communicate with each other when a Torrent is blocked. If the MPAA actually spiders the torrent (rather then trusting information from the trackers), there should be no false edges in the graph. The only collateral damage is if a pair of systems is also performing legitimate communication at the same time they are participating in the Torrent, something the ISP probably considers acceptable.
  
• Any real collateral damage (incorrectly blocking content) AT&T can say is the fault of the MPAA.
  
• It should be robust in the arms race: if the pirated material is open and distributed in a P2P manner, the MPAA's spiders should be able to track it. (Remember, even if CAPTCHAs are used to protect trackers or aspects of the systems, solving a CAPTCHA only costs $.01).
  
• And its inexpensive. All AT&T has to do is deploy a small program to set and release a bunch of router ACLs, and thats it. AT&T can even keep the number of ACLs reasonably low, because they expire quickly and only need to be partially effective. No new hardware is required and everything can be fully automated. All the real costs (of spidering the Torrents, content identification, affirming that it is actually a copyright violation, and constructing the graphs) is placed on the MPAA or their contractor.

Likewise, (IANAL) AT&T can possibly avoid most liability. They aren't doing any wiretapping, nor even making a decision about which traffic to block.

Finally, AT&T has a huge number of reasons to deploy such a system:

• It keeps the content providers happy for when they are negotiating their compete-with-iTunes/Netflix video on demand and cable TV services.
  
• It keeps the content providers from pushing through very draconian legislation, or at least draconian legislation you aren't happy with. (It can F-up your competitors, but thats just a bonus)
  
• And it drops their bandwidth bills by 30-50% by eliminating a large amount of deliberately-noncacheable (both politically and because of bittorrent encryption) traffic.
  

This won't stop closed-world pirates, those with a significant entry and secrecy, but those are far less significant. Closed-world pirates are much lower bandwidth for the ISP, because its far more difficult for pirates to get the content. But it should be able to shut down Bittorrent for open-world piracy, without blocking legitimate BitTorrent. It also won't stop child porn, although AT&T would probably claim that it does.

This was speculation. I have no evidence that this is what AT&T is planning. But given the huge expense (deep packet inspection), legal implications (wiretapping, false positives) and limitations (cryptography), I find it doubtful that AT&T really wants to detect copyrighted material directly. Performing deep packet inspection at line rates, especially to match a large database of copyrighted material, is hugely expensive, and would fail in the presence of encrypted Torrents and SSL-equipped Torrent search servers.

Thus I'm almost certain that if AT&T truely wishes to carry forward with its copyright-enforcement plants, the system will be similar to the one I've described.

Detecting this if they do deploy copyright enforcement would be possible, by participating in torrents (to generate the block) and then checking how that affects connectivity. If AT&T blocks Torrents but other TCP connectivity in those port ranges remains between two hosts, they aren't using only the speculated system, instead they would have to be directly inspecting the traffic between the hosts to determine that an individual flow is participating, information which can only be obtained by directly monitoring communication between the two hosts.

EDIT/addition: Richard Bennet has also discussed this technique at the Network Neutrality forum on 1/26/2008 (Slides at Richard Bennet's web site, on how easy it is to find pirated materials and participating peers to tell the ISP what to block).

He also brings up the important question: "Is there any reason that such an automated system should not be used, or does Net Neutrality now connote a license to steal?" This is a tough argument to counter.

The ongoing discussion can be viewed at The NNSquad Mailing List archive.

EDIT/addition #2: Delayed release of keys (distribute then release keys, as Richard Clayton pointed out) would slow down any spider, but also slows down users from getting content. The spider could still block all users after the key is released, and as people couldn't tell what they are downloading BEFORE the key is released, the MPAA could produce a large number of poisoned (false data) torrents during this window.

Comment spam is worth real money...

(Note: Links are deliberately not clickable, we don't want to give the Spammers pagerank)

Blogger has a pretty significant amount of protection against comment spam. They have to, because comment spam degrades the blog ecosystem. On this personal blog, I've just gotten comment spam like this:

I have to say that I love this article. I have searched for many weeks to find an article about this topic. This blog has been so simple and has a lot more features than other blog articles. The layout and design is great. I will continue to come back here for every articles. Thanks.........

Eva Maryam (a link to kitchencabinets-online.SPAMblogspot.com)

www.jetblue-airlines.SPAMblogspot.com (another clickable link)

Both blogs are simply full of automated-feed content from www.SPAMgetmyarticles.com, a company which gives free articles for posting on websites, probably mostly spamblogs, to make them seem legitimate. And the rest of the blog is a pile of Google adds.

It cost the spammer roughly $.01 to post that single bit of comment spam!

I have "Word Verification", aka the comment CAPTCHA, turned on. This means the spammer either had to invest a lot of effort in building or buying an automated tool, did it manually herself, or outsourced the CAPTCHA solving to a human, such as the Amazon Mechanical Turk, a porn-for-CAPTCHA service, or a Chinese Turing farm. But in any case, all the alternatives effectively cost real money. A CAPTCHA can protect something valued at less that <$.01 or so, but anytime the value is >$.01, CAPTCHAs are useless because you can always just hire people.

I should consider it a complement that the spammer would spend $.01 to post an advertisement on my inconsequential blog. And thus I can see how the spam blogs make money: there are a lot of adds spewed out on that page and it costs nothing to set up. Just a single click might make the spam-blogger $.10 or $1.00 or even more.

And they make enough money to make solving CAPTCHAs worth it, which means blog spam is far more valuable than email spam. An interesting result, and not good for the viability of blogs when comment spam on a random, nearly unread blog is actually worth money to the spammers.

The best counter is probably to attack the add-blogs themselves. All of the content in the spam blog itself doesn't cost the spammer, but by not paying for hosting the Spam blog, they are vulnerable. If Google actually responds to my flagging of the spam blogs which post comment spam, this would disrupt the spam-blog ecology.

But we will see in the future whether this happens, whether Google decides it benefits more from add impressions through spam blogs or is hurt due to the disruption of the blog comments system. I hope for the latter, as manually removing such spam costs me more that $.01 in my time.

The interesting thing is Get My Article's business model. Its free to use the content, but submitting the content requires paying $20/month, and requires actually creating semi-real content! So why are people paying good money to have their legitimate text articles (albeit complete with links) on people's spam-blogs?

Hofmann's Crash

I was at MotoGP racing this weekend, having fun with my camera. One of the photos I captured was Alex Hofmann's crash during free practice in MotoGP, when he was T-boned by Sylvain Guintoli on the enterance to the corkscrew.

I was using a rental lense on my Canon XTi, with a deliberately long shutter speed (1/200) to increase the sensation of speed and depth of field.

The full sequence is here.

I've released theses photos under a BSD-style (aka Creative Commons style) license. Since I'm just an amateur having fun, it is better to just get my name out there, and to maximize the number of people who go "Hey, thats a cool photo".

How to (and how not to) run an airline

For those who have yet to experience the joy of East Coast air travel, there is one bane beyond all others: East Coast thunderstorms. During the late afternoon and evening, masses of thunderstorms often form, blocking airports and flight paths from Boston to Washington. These storms often create "creeping delays", where all Air Traffic Control can tell the pilots sitting on the ground is "ask again in half an hour", because it could be 15 minutes and it could be 4 hours before the planes can fly again.

In two trips within three weeks, I got to experience this first hand.


The most recent, from Washington DC home, was on JetBlue. With a tight travel budget and DC's outrageous hotel costs, an extra night and a morning flight was not in the cards. So on the evening of June 28th, I arrived for the 9:20 PM flight from Dullis to Oakland, Jet Blue flight 321. Sometime around 6pm, the airport basically was shut down: thunderstorms were blocking all routes east and north, and a storm was heading directly to the airport itself.

During this time, other airlines still boarded people and shoved planes onto the tarmac. JetBlue did not. The counter personnel said "we don't want to board the planes until we know you can take off, its more comfortable sitting here". An airline than learned its lesson the hard way.

More important, they communicated with the passengers. Every half hour or so, the counter staff would check in with a pilot on a plane and get an update from Air Traffic Control.

One pilot (the pilot for my flight) stayed at the counter and helped out: explaining to people the cause of the delay, looking up flight status on his smartphone, showing the weather radar to people, assuring us that he was NOT going to cancel the flight to Oakland, and even detailing the tricks he was going to pull to try to get us out as promptly as possible, a scheme which required shanghaing off-duty and over-houred flight attendants to board us 15 minutes before our scheduled cabin crew was due to arrive from a connecting flight.

One of the counter staff even unloaded a few drink and snack carts from the plane, with a "I know this won't make you feel better, but it makes me feel better, so help yourself". The good customer service even continued onboard, with the pilot unlocking the pay-per-view movies.

So although the flight was almost three hours delayed leaving (but only slightly more than two hours arriving, the Pilot put the pedal to the metal), and other flights suffered even longer delays, the process went as smoothly as could be expected.

About the only thing which would have improved the situation would be a weather and/or weather + air traffic display in the lounge, so the customers could see for themselves the airborn mess.



This was in sharp contrast to United flight 19, on June 8th. A similar evening flight, from JFK to San Francisco. Weather was moving in, and any pilot worth his salt would have seen the impossibility of getting off the ground. Nevertheless, we boarded the plane on time.

It turned out June 8th was going to be a ClusterF*** of a flying day out of JFK. Earlier in the day, Air Traffic Control on the east coast suffered a major computer crash. One of the two taxiways at JFK was closed for construction. The East Coast Thunderstorms made their appearance. And an emergency landing on one of the other runways.

But the United pilot told us nothing, simply moved the plane to the taxiway and parked it on the side. No updates, no reports.

The only reason I knew about the weather issues (and resulting routing issues), the emergency landing on the other runway, and most of the other problems was because the pilot did not turn off the ATC channel on the entertainment system, so I listened away to 'xxx, switch to controller C, wait for him to contact you, it will be a while' and 'all emergency vehicles roll to runway Y'.

Even the cabin crew didn't know about what was going on, relying on me to relay information to them! After an hour or so, they distributed cups of water but provided no other cabin service while we were sitting on a taxiway with the engines off.

Four hours later, we were finally in the air. Again, I knew about departure information long before the cabin crew was informed, let alone the passangers. Even during the ascent, the frustrating lack of communication continued, with the pilot detailing to ATC the "moderate" turbulence we were passing through but saying almost nothing to us poor souls along for the rather bumpy ride.


So thus ends the simple lesson in how to, and how not to, run an airline.

iPhone Redux and the Left Turning Porsche...

OK. I still don't like the lockin policy. I find it horribly objectionable.

But I got a chance to play with an iPhone this weekend, a good 10 minutes of lustworthy exploration.

Yeah, the edge network is sucky, but it is sufficient for a lot of work.

Yeah, the lockin policy is repulsively crippling.

But the thing is so well done, so well put together, so easy to use, with all the little touches, that if I didn't have 7 months to go on my cellphone contract, I'd go out and buy one today.

I'll still probably wait (I don't want to spend an extra $150 bucks to get out of my cellphone contract), but the iPhone looks seriously worth it.

iPhone Lockdown and Intent-Based Pricing

There are several applications I'd want to run or port on an iPhone. This includes a full ssh environment, subversion version control, and some custom scripts using ImageMagic which would allow me to process, manipulate, and upload photographs using my digital camera (assuming you could adapt the iPod port to a camera or compact flash card) : all tasks I perform on my Mac laptop but which would greatly benefit from the greater portability of an iPhone.

Yet Apple and AT&T's lockdown policy, only Apple authorized applications can run on the iPhone, means I will be unable to use the iPhone to its potential. I understand the reasons why Apple and AT&T want this property: they want to limit applications which can run because they wish to bill for service based on intent.

At $10 for 1500 SMS message at 1 kB/message, SMS messages are worth roughly 1.2 Mb/$. With voice (beyond the first 500 minutes) at roughly $.05/minute and approximately 8 kbps, vocie is roughly 10 Mb/$. Finally, at "unlimited" data (with a reasonable limit of say 5 GB) for $20, the data traffic is 2000 Mb/$. Thus the intent of the bits, whether it is an SMS message, voice, or best-effort data, effects how it is billed. Thus AT&T's interest is to ensure that the iPhone can't circumvent intent-based billing.

Overall, there is a design philosophy which is creating a sealed box rather than an open box. The sealed box offers some better security properties (as AT&T theoretically does not have to worry as much about misbehaving iPhones), but the security properties are somewhat illusionary. Attackers will still be able to compromise the Safari implementation and gain control of iPhones. It will be difficult for attackers, but doable and highly attractive.

Additionally, the hole in the sealed box, the ability to run sanboxed Ajax-ish web applications, defeats AT&T's intent based pricing, the stated and implied security goals, and Apple's stated goal of a pristine user experience. An Ajax-ie webpage could easily interface with IM protocols, replacing high-value SMS traffic with lower value bulk-data. It is vulnerabilities in the web browser which attackers will exploit. And the interface will never be as good as a native interface running directly on the iPhone.

In the end, the iPhone is a porsche which can only turn left.

If you only ever want to do what Apple has decided you should do (namely email, web surfing, music, and a phone), it is a beautiful platform, and probably worth every penny.

If I could obtain development tools and install new applications, I would buy one in a hot second, even with the transition costs as a Verizon customer.

But with the current model of a sealed box, I will not buy one and will urge my friends and family not to buy one, at least until it costs no more than a basic phone. It may be beautiful, but it is crippled.