Apache attacked by a "slow loris" [LWN.net] (original) (raw)

We're bad at marketing

We can admit it, marketing is not our strong suit. Our strength is writing the kind of articles that developers, administrators, and free-software supporters depend on to know what is going on in the Linux world. Please subscribe today to help us keep doing that, and so we don’t have to get good at marketing.

The slow loris is an exotic animal of southeast Asia that is best known for its slow, deliberate movements. This characterizes the technique used by a new Denial of Service (DoS) tool that has been named after the animal. Slowloriswas released to the public by security researcher "RSnake" on June 17. Unlike previously utilized DoS methods, slowloris works silently. Still, it results in a quick and complete halt of the victim's Apache web server.

The release of slowloris was only done after RSnake had contacted the Apache security team. Their response, while quick, was not quite what he expected:

RSnake commented that this response misses the point completely and that the security tips advertised are of no help. Subsequently, he released the slowloris script, which was followed by a confusing discussion that ranged over multiple blog postings, comments on the postings, as well as various mailing lists. On one side are those hard-boiled experts that say they knew about this technique for years and that it is nothing new. On the other side are those who think this is genuinely new or at least new to the public, and that it could have a devastating effect on the internet as a whole or at least on the half of the world wide web that runs on Apache. Another Internet Storm Center (ISC) post provides more context, along with some useful comments.

However, the majority seems to be stunned by the simplicity of the attack and the fatal effect of it, as well as being puzzled by the reaction of the Apache security team. The team's response makes it seem as if the slowloris attack is well-known, leaving Apache installations vulnerable to DoS by script kiddies, and that there is nothing the Apache developers can do to prevent it. Consequently, they also closed the bugzilla report.

One particular commenter expressed his concern on the full disclosure mailing list with the following words:

I'm out of doobies, and i get nervous when i read lines like this :

AFFECTS
Apache 1.x, Apache 2.x, dhttpd, GoAhead WebServer, Squid, others...?

NOT AFFECTED
IIS6.0, IIS7.0, lighthttpd, others...?

It is not entirely clear which web servers have the means to defend against the attack, but there is general agreement that there is no way for Apache to completely defend against it, and that IIS is not vulnerable to the slowloris technique.

Looking more closely at the slowloris script provides an overview of the technique used. Slowloris gives the attacker a simple way to open an HTTP session with a server and to keep it open for a very long time — a lot longer than it would usually stay open: minutes or even hours.

The way the script achieves this goal can be likened to a person at a checkout lane in a store. Everyone has encountered someone paying the cashier, one by one — literally — in pennies. This can take time; often it feels like it is taking forever.

To the company that has a chain of several stores, this random person does not affect its business. For an online shop, however, there is a single URL and slowloris unleashes hundreds if not thousands of these people approaching the checkout lane with an endless supply of pennies ready to block the queue. For HTTP, slowloris uses HTTP headers instead of pennies, and it is ready to add a new header every 5, 10, or 299 seconds.

Unfortunately, the Apache at the cashier has no memory. With every penny dropped in its hands, it resets the timeout counter. With this technique it is rather simple to block every server thread or prefork process and bring the web server to a complete halt. Because the default timeout setting for Apache is 300 seconds, each header added can stretch things out for that much longer.

An unfortunate side effect of this attack method is that the access log of the web server will not show it is under attack. Also, the messages in the error log are likely to be sparse. The CPU will be idle, no disk IO will be done, and there will also be hardly any network traffic to be seen. All you can observe is a large number of open network connections in the ESTABLISHED state.

Obviously, this is an application level attack. In their book on Internet Denial of Service, Mirkovic/Dietrich et. al noted that application level DoS is difficult to handle: "[...] many defenses are not able to help you defend against this kind of attack".

So we are back to what the Apache Security team concluded: This is an inherent problem for servers. If you want to serve, then you have to accept clients, and, if they intend to block you, so be it.

But, let's not give up so fast. Obviously, if the well-known proprietary alternative from Microsoft, IIS, is not affected by this problem, there are other solutions. What IIS does differently, is in the way it handles incoming requests: There is no static tie between a worker thread and a network socket in IIS. Rather, the workers are organized in a pool where they wait for incoming TCP packets (rather than TCP connections as Apache does). These packets are then assigned dynamically to threads. So, an idle connection occupies a socket, but it does not block an entire thread. Thus the web need not be shut down by penny-wielding customers or slowloris.

Some developers pointed out that the AcceptFilter directive used in conjunction with FreeBSD's kernel http_accept filter should be ported to Linux in order to help with the defense. But, the http_accept filter only works with clear text traffic, so this defense will not work for business-critical services running over SSL (i.e. HTTPS).

It has already been noted that you can catch the single attacking IP address with netstat and block it via your firewall. Or you can use any of the Apache modules that limit the number of sockets allowed for a single IP address. Mod_qos seems best suited for this purpose. But this could block proxies or NAT routers that bundle multiple clients onto a single IP address. A threshold of 30 or even 100 sockets is nothing that should pose any problems to the clients behind these proxies, unless those sites are truly huge. Limiting connections to some threshold would help guard the server. But slowloris is only a proof of concept of a HTTP DoS. It could easily be extended into a HTTPS Distributed Denial of Service (DDoS) attack of a far nastier nature.

So it seems slowloris does not use the full potential of the attack method. The method can also be used more generally, which is why the name does not quite fit. The slow loris is an exotic animal, but the technique the script uses is not really exotic. It is very natural indeed: it attempts to delay an attack from the client side like slow internet connections used to do. So, if there is a term that describes the general form of the attack, then it should be "Request Delaying Attack".

In a follow-up blog post, RSnake has extended the concept to use keep-alive requests, and then delaying any subsequent requests. Other techniques are possible. They are described in great detail in a message to the ModSecurity mailing list from 2006.

Still, the attack has not yet been publicly observed in the wild, and there are still many experts that consider DDoS a non-issue.

RSnake did not claim to have invented the idea, in fact he points out that Ivan Ristić had described it in his book on Apache Security in 2005. Furthermore, posts in the various mailing lists suggest the concept has been around since the 90s, but RSnake has given this simple technique a wider audience.

Now that there is an audience, it is disturbing to see that the Apache community has so little to say about possible defenses. There has been somediscussionon how to handle it, but overall the market leader seems rather complacent these days.