Friday, April 5, 2013

IT Pro confession: How I helped in the BIGGEST DDoS OF ALL TIME


Spamhaus is a Dutch company specialized in gathering lists of spam sources on the Internet and publishing them. Many mail sites - mine was as well until I moved it to Google - use their service to drop the spam before it even hits the mailbox. So, positive.

Enters another player, CuberBunker, another Dutch company, specialized in hosting pretty much anything except "terrorism and pedopornography." At least that's what it claims.

Spamhaus volunteers started detecting spam being sent from the CyberBunker servers, and they proceeded into blacklisting them. That started the beef between the two. You will find the CloudFlare blog entry here, CloudFlare being Spamhaus's hosting company.

CyberBunker clients, instead of doing something technically clever, ran a script-kiddie type DDoS: a massive attack using DNS amplification. As someone put it, that was akin to trying to take a person down by using a machine gun in a crowd.

This was made possible due to the large number of DNS accepting recursive resolutions from anywhere on the Internet.

Trevor Pott, a sysadmin, took the courage to write a short article for The Register on how he was an unwilling help for the CyberBunker side: he explains the mistake he made and how he corrected them. He also has a few cool tricks, some of them helped in limiting the damage. Even the US-CERT has a document on securing the common DNS servers. Oh, this document was published in 2006 ... the problem is nothing new. Some more information and stats can be found on OpenResolver.

Bottom line:


  • Sysadmins tend to forget about the "small" services that keep the network running: NTP, DNS, DHCP and so forth, and often, these servers are left unpatched or untested;
  • That sucks, but different version of the same software may have different default configurations! The keyword is "inventory", and knowing where each component of your infrastructure runs. And reading the release notes;
  • There is no one-magic-bullet-security-stuff that will solve all the problem: Trevor had a few layers in place that mitigated the damages caused by the misconfiguration. Namely, he limited the bandwidth available for DNS to 500Kb/s. He initially saw a 10Mb/s surge;
  • If you have people like Trevor who understand what they are doing, your network is in good hands.