Aug 20
Security

Dan's talk at Black Hat on 'The DNS Bug' aka CVE-2008-1447 was packed. By this time I had worked out that BH attendees, much like Catholics fill up from the back, so you can usually just walk to the front and find a seat. I did and ended up three rows away from Dan and his podium, a decision I later regretted.

It's a long entry, so I've bold'ed the key parts of my rant.

With all the hype, my plan was initially to not attend Dan's talk and go to something more interesting. My next thought was that most of the talk get completely dissected after BH anyway, so I would rather take advantage of the 'being there' factor and go watch the soap opera unfurl live.

He started off pretty well, gave some insight into how he had discovered the flaw, how the sooper sekret multi-vendor discussions had gone etc. He pointed out that Pieter de Boer was the first to 'rediscover' the bug 51hrs later. He also showed an awesome video of the patch rates around the world.

I was interested that by speaking to Trustwave, the fear and loathing of PCI was used to drive many corporates into patching.As Dan pointed out 'probably the first time compliance has been used to accomplish anything useful.' I'd restate that as 'probably the first time compliance has been used to drive rapid response to a tangential security incident.'

After that things went downhill. Dan decided to use the rest of the talk to provide illustrative situations of what a MITM can do. He went on at length describing all of the various protocols, services etc. that can be MITM'ed. He also pointed out that bugs don't need to be 'sploited in isolation which gave him more time to go into scenarios. He could have used this time to go into more interesting discussions around the usability of DNSSEC, practically solutions, more details about the bug etc. Anyone who knows what a MITM is should have been bored.

During this time I started to get irritated, but I mean really physically irritated. I couldn't work out why, until I realised it was some element of his stage personality that was pissing me off. At first I ignored this as me just being judgemental of a socially akward geek with a podium and media; and chided myself. However, after unpacking this in rants at anyone who would listen during the course of BH and DC I realised what was bugging me.

Dan was show boating. His entire approach to the thing was show boating. His 'wait for blackhat' plea, which irked me because it was just too obviously a media stunt combined with his 'ra ra, hand wave' presentation made it very clear. What's worse, his claim that he was trying to help people and get things fixed ASAP was hurt by his method of disclosure. By the end of the talk, I had switched to red pen (I wan in rant mode) to document why I felt his disclosure method was at fault for the 90 day patch window.

There are two things required for patches to be deployed, a working patch, and a willingness to install the patch. To drive the willingness, security people (or admins charged with patch responsibility) need to understand the risk so they can make a proper risk decision relevant for their circumstance and organisation. Dan's work to coordinate the release of the patch was great, we got multiple patches all at once without in-the-wild exploits. However, how urgent was this patch? Should we divert resources from existing stuff to test and deploy it? Is there a significant risk difference between external DNS and internal DNS etc.? The delay in patching NAT'ted DNS servers illustrates how this lack of information hurt patching. When a client phoned me to ask how urgent the patch was, all I could say was I'm not sure and to include it as part of the normal test/deploy cycle. If we had this information, if the security community had it, we could have properly understood the bug and known what the risk was.

This does make Dan look a little hypocritical. The reason I choose hypocrisy and not 'a failure of unintended consequences' if for two reasons. The first is that Dan actively crusaded to keep discussion off public lists, this goes far beyond whatever NDAs he was probably slapped with. When one hippy posted to full-disclosure with a hypothesis, Dan personally mailed him and asked him not to discuss this stuff publicly. While it is silly to tell one of the most inquisitive communities in the world not to talk about something, it is antagonistic to the goals of 'get everyone to patch' as well (because it prevents an intelligent risk-based decision from being made). I also don't think Dan has an excuse for not understanding models of disclosure and their implications. We've seen vendors semi-secretly patch bugs before, and we know the outcomes, and the debate is a couple of decades down the line. Finally, the reason I think Dan should take responsibility for this criticism is because he made such a big play to put himself in the centre of it.

I'm probably being a little unfair to someone who spent a lot of time working on something and who in the end got the world a little safer (and who I have never met personally). So take this as a commentary on his methods and media image, and not him personally. Ideally, take this as a lesson of how not to disclose bugs, ever.

Posted by Dominic White

Last modified on 2008-08-24 21:08

10 Comments

Display comments as(Linear | Threaded)
  1. vpr says:

    Given how things unfolded, how should he have played it? PS I think he had his thunder stolen.

  2. Dominic White says:

    He should have provided a description of the attack (not example code) when the patches were released. No assuming we only care about finding and fixing vuln, that's where the discussion ends. If we are interested in getting Dan some publicity he could have promised a full overview of the behind the scenes stuff at Black Hat (secret meetings at Microsoft with the architects of the internet sounds cool enough), and maybe even some example of the attacks he talked about, instead of just talking about them (e.g. Evilgrade), if he wanted to demo an attack people hadn't seen. But, he had more than enough cool stuff to talk about with the DJB links, the vendor coordination, the patch window etc. I'm not sure his thunder was stolen, he packed both the BH and DC talks and got more media publicity than most (he even got the Pwnie ;) ). But the real question is, do we care about his 'thunder', and should disclosure models take into account providing 'thunder' for researchers as a reward?

  3. Dan Kaminsky says:

    I'm a little confused. First you don't need to hear exactly how important MITM attacks still are in 2008 -- you know that, for example, reverse DNS allows arbitrary TCP/UDP behind the firewall via Java, which chains really nastily with the SNMPv3 bugs. You know that email is completely hosed, that this facilitates perfect spear phishing attacks for code execution, and that this in fact creates an auth bypass on almost all major web sites. You know that there's an entire world of networked applications that aren't web browsers that are about to be open to attacker fuzzing. You even know that SSL, the gold standard in technologies that's supposed to protect us from a DNS bypass, is constantly being abused, that certificates are rarely being checked by non-browser applications, and that the browser experience is plainly insufficient. You know all this, and you still tell your customers to follow their standard patch cycle? Look. You didn't know. It's OK. If Halvar Flake, who's smarter than all of us combined, thought that we had enough other layers of defense to protect us and was wrong, then it's OK for everyone to have not known. That's the point of research -- to find new things, and figure out what to do about them. Here, there was a situation where there were some pretty new (or, frankly, insanely old) and nasty things found, and while not everyone would patch without knowing exactly what they were -- some people would. More than none. Ended up being, way more than I thought. Hundreds of millions of people were protected by my approach -- and, in the end, I still disclosed. You can complain all you like and I'm just going to look at that 120 million number with a big goofy smile on my face and be ecstatic. That's so cool! And, even better, this research is finally out. We can all figure out, as a community, what we should really do about DNS, and how we can finally start actually encrypting and authenticating our traffic both inside and outside our perimeters. It's 2008. Can we have secure email now? Please? Certainly it's a more interesting question how we can do that, versus whether I should have just showed up on August 6th, patches in hand, and delivered the speech as I did.

  4. Dominic White says:

    No ways, you responded, that's pretty cool, thanks. I agree that you needed to drive home the importance of what this sort of universal MITM allows, and that some time should be spent doing that, but I don't agree that it should have taken up so much time from your talk. But that sounds like a point of preference and not fact, so I'll concede it. But I explicitly agree that I and other (including bright ones like Halvar) didn't get the full implications of the patch/problem, and hence couldn't make a proper decision. That's sort of the core of what I think went wrong (but I agree your multi-vendor patch-fu was awesome). I really like that you care, and that you wanted maximum goodness by getting lots of people patched, and that's why I think it's important that we learn from this type of disclosure to make even more goodness happen next time, and why you would be interested. I also agree we need to do better, that we need some way to get rid of or fix these old insecure ways of doing things without relying on the whole world to agree or willingly change (see DNSSEC or even DJB's 2002 rants on randomness), but I don't think this is the way to do it, the media-fatigue would make people stop listening. We need a better way, and 'responsible disclosure' [for whatever middle ground you take that to be] is one of the building blocks, don't kill it.

  5. vpr says:

    This is turning out to be more fun than expected :) Disclosing flaws is always a difficult task, and you inevitably upset someone. Dan did the *right* thing for the right reasons. Given that the good guys and bad guys don't have the same resources, I think it's perfectly legitimate to use the media / PCI to *help* drive the change.

  6. Dominic White says:

    I'd argue Dan did a good thing (re disclosure), and my point is how to do that good thing better. Using PCI and the media to help drive the change is fine, if fact I applaud it. That's not the same as my personal dislike when someone *also* clearly uses the media for personal promotion (subtly is okay :) ). That's a point of preference though. Glad you started commenting :)

  7. Dan Kaminsky says:

    For the record, the only people *I* tried to promote were the vendors -- rather than feed me a line about "we'll fix it when we get around to it", they turned on a dime, showed up in Redmond, kept complete operational secrecy, and pushed out a synchronized fix. All I did was find the bug, and help decide how to fix it. I know it seemed strange that I spent so much time on the impact of MITM. You have to remember the context -- the bug was out, and people were *still* claiming it didn't matter. You can actually find them, these huge position pieces saying of course everything is already armored against such a trivial vuln. They were all very, very wrong -- but heh, I wanted those thirty days, and wasn't about to correct them. Yet. Here's the thing about responsible disclosure: I'm a huge fan of it. It's what I've been doing for almost a decade. This bug was a little nastier than my normal fare, though. Even with all I'd done, getting the synchronized fix, I'd spent enough time in corporate environments to realize that: a) Given full details, this bug could be weaponized in minutes, and b) People couldn't even *find* the servers they needed to patch in 24 hours, let alone get everything tested and deployed These are ultimately proven facts, backed up by patch rates in the field and exploit development speeds once minor details came out. So, given these facts, if I'd come out, no warning, on August 6th with my talk exactly as delivered...how do you think it would have been received? Seriously? "Why didn't you tell us a couple weeks ago! We could have patched! Obviously you just wanted to protect your 'thunder' at Black Hat!" It's what I thought would have happened -- and I thought it would have happened against a backdrop of exploitations in the field. Now, we don't know for sure what might or might not have happened. But let me ask you: I *did* get a lot of vendors spending a lot of money and expending a lot of effort to protect their customers. How could I ask this, and not at least offer to do what I could to also help? Regarding the community ask: One guy, who I hadn't even heard of, figured it out in 51 hours. What would it have meant for me to say nothing, if someone else from the community were to spill it all immediately? I had to at least try. It didn't need to be forever. In fact, if I'd asked for months, it would have been out in hours. But I asked for a few weeks...and for the most part, I got it. And look. More people were patched, less people were hacked, and open and full disclosure of security findings is widely seen to have helped rather than to have hindered protecting the network. That's how things should be.

  8. Dominic White says:

    Dan, you've convinced me. Certainly, that I was wrong about a lot of what you did. I think there's room for further debate on the theoretical disclosure stuff, but that's not that useful and it's probably only because I wrote a thesis on patching. Nice work, and thanks for taking the time to reply.

  9. Allen Baranov says:

    Hi, My 2c. There should have been some sort of semi-disclosure. Having said that I think the amount of disclosure that started to come out unofficially was pretty good. Before Dan's talk I knew the following: - Some DNS Servers don't randomise their ports. - There is a new way to speed up the guess work for the key that the servers use. - Random ports will sort you out. - There are patches available. - You need to use these or your DNS can be poisoned. I really didn't need to know *how* the speed-up was done to correctly assess my security risk and that was all I got out the talk.

  10. Dominic White says:

    Hello Allen, After 2002 (and DJB's port randomisation paper) you could have know: - Some DNS Servers don't randomise their ports and random ports will sort you out. - There are patches available and you need to use these or your DNS can be poisoned. This part (and the patch) was nothing new. There has been *no* speed up in guess work for the key. This is a straight forward brute force of all TXIDs. Proper disclosure may have helped you better understand that. What changed is the addition of the RR vector to change the cache poison from single-shot to persistent. In short, your understanding of the problem is exactly the sort of misinformation that resulted from this sort of half-cocked, vendor-aligned disclosure.

Add Comment


E-Mail addresses will not be displayed and will only be used for E-Mail notifications.

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA