Two of the largest distributed denial-of-service attacks ever recorded on the internet have been launched over the last seven days, taking advantage of sloppy configuration settings to force prominent websites and their protection services to fend off massive amounts of malicious traffic. And this just might be the beginning.
Both the Github attack last week and a similar-but-larger attack against an undisclosed company identified and prevented by Arbor Networks on Monday took advantage of memcached servers exposed on the public internet. The attack detected by Arbor Networks had a peak traffic load of 1.7 terabits per second, ushering in what it called “the terabit attack era” in a blog post Monday.
“While the internet community is coming together to shut down access to the many open memcached servers out there, the sheer number of servers running memcached openly will make this a lasting vulnerability that attackers will exploit,” Arbor’s Carlos Morales wrote in the blog post.
Memcached servers allow applications that need to access a lot of data from an external database to cache some of the data in memory, which can be accessed much more quickly by the application than having to travel out to the database to fetch something important. Companies use these servers to speed up page load time and deal with spikes in demand, and they have been widely used across the internet over the last decade or so.
Usually, these types of servers are used internally, disconnected from the public internet but accessible within a trusted network to improve internal application performance. But it appears lots of people have been leaving memcachecd servers exposed to the open internet, where they can be discovered and exploited by just about anyone.
A big part of the problem is that until very recently, memcached servers had the UDP port open by default, according to Cloudflare’s Marek Majkowski, who outlined the basics of this attack structure in a blog post last week. UDP (user datagram protocol) is a basic part of the structure of the internet, like the more well-known TCP (transmission control protocol), and it was often used with memcached servers back in the day because it was simpler and faster than TCP. That’s not really true these days for a variety of reasons, but there are still lots of memcached servers using UDP scattered across the internet.
Brad Fitzpatrick, long-time Googler, Seattle resident, and the creator of the memcached open-source project, said in an email Tuesday that the original version of memcached did not support the UDP protocol. Facebook contributed code in 2008 that added that support, but without authentication, under the assumption you’d still run these servers inside trusted networks. A later version of the software released long after Fitzpatrick was no longer involved with the project added authentication support for web-facing TCP protocol users of memcached, but UDP was left hanging in the wind.
That is, until last week. After word of this new attack method began to spread, the current leaders behind the memcached open-source project released a new version last week in which the default setting locks down the UDP port. This can also be done manually.
There are thousands of memcached servers out there with unprotected UDP ports, according to Cloudflare. It always takes time for people to update their servers when problems like this are detected and analyzed, but at least in this case there are precautions that can be taken, unlike the Mirai botnet powered by a bunch of dumb connected cameras that can’t easily accept software updates.
Still, expect to see more of these attacks in the near future until enough folks update their servers to deny malicious hackers an easy attack vector. Cloudflare is also urging ISPs and networking companies to come up with better systems to detect IP spoofing, which allows attackers to pretend to be someone they’re not in order to launch these attacks.
