On Friday, October 21 an exceptionally large attack was launched which affected a huge number of sites on the Internet. Some prominent sites were affected including PayPal, Amazon, NetFlix and Twitter. No sites we host were affected, although some sites using PayPal as a payment gateway did have some trouble.
To get some sense of the scope of the attack, you can take a look at downdetector.com and Threat Post. This incident has yet again pointed out that security on the Internet is a mess.
You may have unwittingly participated!
As is usual with slightly technical information, the news media reports of the incident were roughly 75% factually inaccurate. It is exactly the general ignorance of security which enables this kind of attack so this is unfortunate. It’s not difficult to understand what happened and how you may have participated. If you are among those people who often use the acronyms, DR, TL, get a clue. Didn’t Read; Too Long isn’t a reasonable attitude in this case. If the Internet is to continue to be useful, we all need to take some responsibility for it. That’s just how it works.
How the Attack Works
When your computer (or phone or any Internet capable device) wants to connect to a web site or other service on the Internet, the first step is to get the numerical address of the service. It’s basically a 3 step process. It gets the address by using what is called DNS, Domain Name Service. There is a centralized repository which exists for the purpose of translating domain names (paypal.com, netflix.com, amazon.com, etc.) into numerical addresses. The first step is to query the repository for a list of servers which have the wanted address. In step 2, your device uses the domain name to connect to one of those servers and get the numerical address (called the IP address). The last step is using the address to connect to the service.
In many cases, the news media reported that web sites and services were hacked or were not functioning. This is sloppy reporting and is not correct. Sites and services were not even attacked. What was attacked was DNS service. This means for example that while NetFlix was up and running with no trouble, a lot of people couldn’t find out where to connect to it. They couldn’t get the address using the DNS system.
As I write this, a very large DNS service provider dyn.com continues to be under attack. For many sites a quick fix was put in place, adding DNS service based with other providers. The DNS system is very resilient. When your machine can’t get an answer from one DNS server, it will try the next and the next until it either gets an answer or the list of servers to try runs out. By extending their DNS server lists, many sites and services were able to quickly restore availability. But that isn’t the end of the problem.
The attack is a distributed denial of service attack. Essentially, the service provider is flooded with so many requests for service that it can’t answer them all quickly enough. The usual way to deal with attacks like this is to identify the source as an attacker and then ignore any further connection attempts from that address. The problem is volume. Each connection attempt has to be read at least to the point where the source address is found and that address compared to a list of addresses to ignore. This takes resources. As the list of blocked addresses gets longer, the time it takes to check it also gets longer. For technical reasons, when you double the size of the list, the time it takes to do a lookup more than doubles. The increase is at best logarithmic.
In a DDOS attack, resource availability at the service provider is usually the biggest problem, but it can get worse. As it was in this case, the volume can be so great that there is not enough space on the wire (bandwidth) for all the data coming in. There isn’t enough left to use for answers. The service is hosed.
This attack is being perpetrated by what is called a bot net. The participating devices are running compromise software called Mirai. A bot net is a collection of compromised computers being controlled remotely and acting in concert. By some estimates, more than a million devices are compromised in this way. This is the distributed part of the attack. By instructing potentially tens of thousands of devices to do DNS requests at the same time and keep doing them, huge amounts of traffic can be generated.
Who is Behind the Attack
No one knows who is controlling the botnet(s) doing this attack. Some news reports are quoting “authoritative” sources who are giving definitive answers. It doesn’t take a great depth of technical understanding to see that there can’t be a general definitive answer. Anyone who says otherwise is unequivocally wrong. There are sometimes some clues and the dumbest botnet operators get caught. The smarter ones use a chain of compromised computers to issue their orders. You would have to solve every compromise in the chain to lead back to someone. That’s nearly impossible.
Some people have been making the case that attacks like this are state sponsored. That may well be part of a larger picture. The article contains some misinformation, but Bloomberg has some theories and a bit more analysis.
Some security people such as Brian Krebs believe that this problem is just getting started. If you can’t pull up that link, the likely reason is that it is currently under attack.
The outlook for stopping this activity in the short run is bleak. There are simply too many vulnerable devices and the compromise is too easy to pull off. Your local service provider is probably failing to do their part. At Deerfield Hosting we have long had countermeasures in place. We often speak with customers who have, unfortunately, been affected by them.
If you have a vulnerable device (such as a router or a security camera from Best Buy), complain to the vendor and manufacturer. If you have a device which still has the factory password and you can change it, change it! The Internet is a peer to peer network. That means almost no matter where you think you are, you are everywhere. So are the criminals. They will find it.
It wouldn’t be out of line to complain about the sloppy reporting. Service providers (like Time Warner and Comcast) generally make no effort to block outbound attack traffic, inform customers that they are compromised or help their customers avoid getting compromised in the first place. They should be doing these things. They can afford it. It’s cheap. It wouldn’t be out of line to complain to them that they are not.
The problems are endemic to the entire Internet and need to be addressed broadly. That means all of us.
Expecting someone else to keep us safe is not realistic. It’s no different than not leaving the keys in your car when you’re out shopping. It’s common sense.