Biz & IT

How Google fought back against a crippling IoT-powered botnet and won

Behind the scenes defending KrebsOnSecurity against record-setting DDoS attacks.

Dan Goodin – Feb 2, 2017 2:12 pm | 76

Credit: Alex Eylar

OAKLAND, Calif.—In September, KrebsOnSecurity—arguably the Internet’s most intrepid source of security news—was on the receiving end of some of the biggest distributed denial-of-service attacks ever recorded. The site soon went dark after Akamai said it would no longer provide the site with free protection, and no other DDoS mitigation services came forward to volunteer their services. A Google-operated service called Project Shield ultimately brought KrebsOnSecurity back online and has been protecting the site ever since.

At the Enigma security conference on Wednesday, a Google security engineer described some of the behind-the-scenes events that occurred shortly after Krebs asked the service for help, and in the months since, they said yes. While there was never significant hesitancy to bring him in, the engineers did what engineers always do—weighed the risks against the benefits.

“What happens if this botnet actually takes down google.com and we lose all of our revenue?” Google Security Reliability Engineer Damian Menscher recalls people asking. “But we considered [that] if the botnet can take us down, we’re probably already at risk anyway. There’s nothing stopping them from attacking us at any time. So we really had nothing to lose here.”

It took only about an hour for Menscher’s team to arrive at the decision to help Krebs. A much more lengthy process involved actually admitting KrebsOnSecurity into Project Shield, a free service with the mission of protecting news-, journalist-, human rights-, and elections-monitoring sites from DDoS attacks that might otherwise prevent them from publishing. A key requirement for admittance is that the person requesting service proves they have control over the site. Because KrebsOnSecurity was down at that moment, Krebs was unable to satisfy this requirement. Making matters worse, the domain-name system settings KrebsOnSecurity used had been locked to thwart the attempted domain hijacking attacks that regularly targeted the site. That prevented Krebs from showing he had control of the site’s DNS settings.

Once Project Shield ultimately got KrebsOnSecurity back online, it took just 14 minutes for the attacks to resume. The first one came in the form of a flood of 130 million syn packets per second, a volume that’s big enough to bring down plenty of sites, but a tiny drop when measured against the resources Google has. About a minute later, the attack shifted to a slightly more powerful flood of about 250,000 HTTP queries per second. It came from about 145,000 different IP addresses, making it clear that Mirai, an open-source botnet app that enslaves cameras and other Internet-of-things devices, was responsible. The attackers followed it with yet more variations, including a 140 gigabit-per-second attack made possible through a technique known as DNS amplification and a 4 million packet per-second syn-ack flood.

An image accompanying Menscher’s talk. Credit: Google

At the four-hour mark, KrebsOnSecurity experienced one of the bigger attacks seen by Project Shield engineers. It delivered more than 450,000 queries per second from about 175,000 different IP addresses. Like the attacks that preceded it, it posed no immediate threat to KrebsOnSecurity or the Google resources that were protecting it.

The attacks were the most powerful in the first two weeks, but as they continued, they incorporated a variety of new techniques. One, dubbed a WordPress pingback attack, abused a feature in the widely used blogging platform that automates the process of two sites linking to each other. It caused a large number of servers to simultaneously fetch KrebsOnSecurity content in an attempt to overwhelm site resources. Google was able to block it, because each querying machine broadcast a user agent that contained the words “WordPress pingback,” which Google engineers promptly blocked. Another technique dubbed “cache-busting attacks” was also stopped.

The DDoS attacks on KrebsOnSecurity remain a regular occurrence even today, and while some have resulted in brief interruptions so far none have caused sustained outages. Menscher shared the following lessons with the audience, which was made up largely of security-related engineers, technologists, and researchers:

Defending a small site is really hard. All of my experience at Google for years was defending a very large site. If we had an extra thousand queries going through to one of our services, it wasn’t a big deal. But Brian’s origin server could maybe handle around 20 queries per second. We saw attacks of up to 450,000 queries per second. How do you deal with that? It’s a little bit challenging. One thing you can do is you can rate limit the bad traffic. So you have to identify the bad traffic and try to throttle that down. Another thing that helps a lot is you can serve good traffic from cache. This takes a lot of load off the origin server. It also gives you this benefit of even if the origin server is unhealthy, you still have its content cached so you can continue to serve users and there isn’t really a visible outage.

Asked why a sprawling service such as Google is able to defend Krebs for free when officials at Prolexic—an Akamai-owned service with a core competency in DDoS mitigation—reportedly said it was no longer viable to continue its pro-bono arrangement, Menscher said:

There’s a lot to be said for economy of scale. In Google’s case, we’re already serving a lot of properties. By having all of that, it’s more cost effective for us to have a terabit of spare capacity. I would expect Prolexic would also want to have a terabit of spare capacity, but then it starts eating into their spare capacity if… there are two dos attacks coming at the same time.

The ultimate takeaway, Menscher said, is that even at a company like Google where it’s crucial to have no more than five minutes of downtime in a given year, it’s sometimes necessary to take risks.

“I was trained as a physicist, and in physics we’re always trying to figure out how the world works,” he explained. “But you have to ask the right questions. You have to investigate things. You always have to be willing to question your assumptions. DDoS defense is very similar. You can’t just look at the attacks you’re getting. You have to be more proactive and try to attract more attacks and take some risks.”

Listing image: Alex Eylar

Dan Goodin Senior Security Editor

Dan Goodin is Senior Security Editor at Ars Technica, where he oversees coverage of malware, computer espionage, botnets, hardware hacking, encryption, and passwords. In his spare time, he enjoys gardening, cooking, and following the independent music scene. Dan is based in San Francisco. Follow him at here on Mastodon and here on Bluesky. Contact him on Signal at DanArs.82.

76 Comments