Distributed Denial of Service prevention: Current state.

Rick Howard: At around 8:30 p.m. on November 2nd, 1988, a 23-year-old Cornell University graduate student named Robert Tappan Morris released what would come to be known as the Morris Worm, the hack heard around the world.

Unidentified Person: Internet-connected computers began to fail across the country. Once it launched, it kind of goes on and on. The University, the military, defense, they were all suddenly saying, wait a minute. I can't check my email because my computer can't do anything else. There was alarm because we didn't know what was going on. There was some concern that this might be some kind of a military attack on the United States. And it just got to the point where administrators were like, we have to shut down these computers. It was sort of a bunker mentality and that you were sitting here and very much felt like you were under attack. To some degree, we were kind of scared because we didn't know in the next five minutes it could suddenly turn nasty and start removing users' files. The Internet had just been created the previous year, and so nobody really knew what it was. By the weekend, the worst of it had passed. Berkeley was able to release a patch within 24 hours. But people were scared. The GAO had estimates of anywhere between $100,000 to $1 million in damages. It's possible, maybe even probable, that we'll see another attack reasonably soon, maybe now.

Rick Howard: According to the FBI, within 24 hours, 10% of the existing 60,000 Internet-facing computers at the time became incapacitated, resulting in the first recorded Internet-wide distributed denial-of-service event, or DDoS. The Morris Worm marked the first global use of a destructive Internet worm, and it was clear that nobody had anticipated that bad guys would use the entire Internet for malicious purposes. We've come a long way since that first DDoS attack some 35 years ago, and for the most part, the technology required to mitigate a DDoS attack has been around for a while. [ Laughter ] According to Erica Chakowsky, an independent tech journalist writing for AT&T Security in 2020, she says that hackers use three broad DDoS attack strategies and specific tactics for each. They use the volumetric attack strategy to generate massive volumes of network traffic designed to completely saturate the victim's bandwidth. Tactically, they launch UDP and ICMP floods or DNS amplification schemes. Hackers also use the protocol attack strategy designed to eat up the processing capacity of network infrastructure resources like servers, firewalls, and load balancers at Layer 3 and Layer 4 of the TCP IP stack. Tactically, they might use a SYN flood attack, spelled S-Y-N, or a synchronization packet within the TCP protocol to circumvent the three-way handshake process required to establish connections between clients and servers. Or they might use the ping of death, where the data packet within the ICMP, UDP, or TCP protocol contains malicious content causing the victim machine to freeze or die. And finally, they use the application attack strategy at Layer 7 by initiating transaction requests that consume finite resources like memory. Tactically, they might use an HTTP flood attack where they send legitimate requests to a web server that are designed to consume as many resources as possible. For me, though, it's been a minute since I looked at the latest developments in DDoS prevention technology. My assumption going into this episode is that any deployed DDoS tools fall squarely in the bucket of our resilience-first principle strategy. The idea here is to survive a DDoS attack, not prevent them beforehand with some kind of zero-trust or intrusion kill-chain prevention strategy. Let's see if I'm right. So, hold on to your butts. >> Hold onto your butts. In this "Rick the Toolman" episode, we're going to explore the world of distributed denial-of-service protection. My name is Rick Howard, and I'm broadcasting from N2K Cyber's Secret Sanctum Sanctorum studios located underwater somewhere along the Patapsco River near Baltimore Harbor, Maryland in the good old U.S. of A. And you're listening to "CSO Perspectives", my podcast about the ideas, strategies, and technologies that senior security executives wrestle with on a daily basis. Fans of this show know that my best friend is Steve Winterfeld.

Steve Winterfeld: Steve Winterfeld, Advisory CISO at Akamai. I've been with the company coming up on five years.

Rick Howard: I've said this many times before, but he's the guy I'm calling for help when I need to leave the country in a hurry. You know, for reasons. He's the former CISO of Nordstrom, a regular contributor here at the CyberWire hash table. And he's one of two editors of my book, "Cybersecurity First Principles: A Reboot of Strategy and Tactics." And he is the Al Borland to my Rick the Toolman.

Al Borland: Folks, I would just like to be clear that those were Rick's thoughts and not Al's. So send your complaints to Rick at csop@thecyberwire.com.

Rick Howard: As Al said, he's been at Akamai for over five years now. For those that don't know, Akamai is a cloud computing company that specializes in security services and content delivery. They've been offering DDoS protection services for many years. They bought Prolexix back in 2013, one of the original DDoS protection companies that featured prominently in the Cybersecurity Canon Candidate book, "Fatal System Error", written by Joe Menn, about some early cybercrime stories back in the 2000s. So when I decided to get up to speed on the latest DDoS prevention tools, I naturally reached out to Steve. He sent over a collection of Akamai white papers on the topic, and the links are in the show notes. So I asked him to come on the show to unpack some of those ideas. I was wondering how DDoS prevention companies like Akamai, Imperva, Radware, Cloudflare, and many others can determine if a customer is under a DDoS attack or they are just getting a higher-than-usual amount of network traffic. Here's Steve.

Steve Winterfeld: We say DDoS, but it really is a misnomer because when you see it, you'll see things like requests per second. Well, then you know those are going against web pages or APIs. You'll also see bits per second. That means you're just trying to overwhelm the pipeline, volumetric attacks against everything. You might see packets per second. Then you're trying to use the resources on the computers themselves. You're trying to make the CPUs crunch so many numbers that they just fall over. And finally, you'll see queries per second. And when you see queries per second, then it's against the DNS infrastructure. And we have to remember DNS, for somebody as old as you, kind of like the phone book.

Rick Howard: I'm glad you mentioned the phone book, Steve, because now I have to explain phone book to all my listeners because they have no idea what that is.

Steve Winterfeld: Oh, my God. Here we go. More history.

Rick Howard: For all of you youngsters in the audience, back in the day, before we had the Internet and before we all had cell phones, every home, business and school had these things called telephones, hooked to a giant network of physical wires that the phone company ran to all of the buildings. If you needed to call Kevin down the street, you could look up his number in a very thick book called the phone book that listed everybody's name and telephone number in it. According to Jeff Nielsen of the Saturday Evening Post, the phone company in New Haven, Connecticut, issued the very first phone directory on February 21st, 1878. It listed the numbers of 11 homes, 38 businesses and the police department. And it was the first incarnation of what would be the domain name system DNS in the Internet age around 1985.

Jim Gilbert: My name's Jim Gilbert. I've worked at Akamai for nearly eight and a half years now. And my focus has been on DNS, both recursive and authoritative.

Rick Howard: Jim Gilbert is the Akamai product manager for external, authoritative and recursive DNS services. Steve brought him along to keep us both honest.

Steve Winterfeld: Not even close, Rick.

Rick Howard: I asked Jim to provide a simple example of how simple DNS can be. I asked Jim to provide a simple example of how cyber criminals use DDoS.

Jim Gilbert: Ransomware is a good, simple example, even on DNS DDoS. So there'll be threats of a DNS DDoS attack. There'll be a sample attack. And then there'll be a request for payment. And after a certain amount of time, and then after that amount of time, there'll actually be a larger DNS DDoS attack.

Steve Winterfeld: Back in the day, it may have been I would pay $10 to knock somebody off a game. You know, it was very low level. And then over time, it became extortion. You know, it's one of the three major extortions. You have ransomware, holding data hostage and DDoS. And it's kind of that bracket of pay me and this won't happen to you. A couple years ago, the U.S. had the most DDoS attacks. Now it's Europe. And that shifted with actors that are more politically motivated. And I don't call them hackavists, because they're more like privateers. Historically, back in the sailing ship days, if a nation needed more ships, they would commission pirate ships to be privateers. And so this is kind of this hybrid of partly a criminal, partly a nation state actor. And somebody like Killnet would be that.

Rick Howard: Okay. I see that as Killnet as an example. But it doesn't eliminate hacktivism. They might use that same tool too, right?

Steve Winterfeld: Correct. Correct. But they're not tipping the numbers from the U.S. to Europe.

Rick Howard: It's also used as a decoy, right, to launch a DDoS attack over here on this part of the network. And then the super cyber bad guys can go in on another part and steal data or do something like that. How often is that happening?

Jim Gilbert: It's relatively frequent. We'll see an attack on the DNS system followed by an attack on other parts of the infrastructure. And you can see the volumes correlate in time.

Steve Winterfeld: Another thing we put out in a different report that I sent you is that also gives the benefit of overwhelming logs. It makes it harder to do investigation when your logs are overwhelmed, overwritten. That paper we did jointly with Financial Services ISAC, we both wanted to highlight that that is a valid and practiced technique the threat's using.

Rick Howard: Besides overwhelming the logs, if we were noticing that you're being overwhelmed by a DDoS attack, I could see where that would focus the InfoSec team, that we've got to stop that. What you're really telling me here is that there's a good chance that there might be something else going on, so don't let your guard down.

Jim Gilbert: Yeah, that's absolutely right. No doubt, even in the most recent weeks, a large healthcare provider that you know and a large oil and gas company that you know, both had these simultaneous attacks. And the InfoSec teams were trying to manage both of the events and trying to cross-correlate it, and it definitely kept them busier and distracted, for sure.

Steve Winterfeld: In a recent report we put out on ransomware, we also noted that, you know, if you had a ransomware attack, that there's a good chance you're going to have a second ransomware attack within the next few months. So as you're putting out one fire, somebody else is lighting another fire. So you have to be able to do both things simultaneously.

Rick Howard: Yeah, so when you finish the first one, don't think, oh, that'll probably never happen again.

Jim Gilbert: Good to go.

Rick Howard: Well, Steve, you mentioned the ways we can measure these DDoS attacks. You said RPSs, requests per second. We got bits per second, packets per second, queries per second. How are you guys deciding that, oh my God, this is not normal, and then we need to do something? How do you guys do that? Not specifically Akamai, but in general, how does a service like this work?

Steve Winterfeld: We've shown that there are known attack patterns. There are, you know, SIN floods or something like that that is a known recognized attack pattern. What we've seen in the last few years is a few things. One, the increase of Internet of Things has given more bots to create larger armies, to create larger attacks. So the speed of attack has increased. You know, many years ago, it would take hours to build up a long attack. Now it's minutes to go from a low-level attack to a new record. And then the types of attack, it was UDP floods, SIN flood, things of that nature, UDP fragment. And there were probably, you know, a handful of those that were the most. Now those handful are less than 50%. You know, it used to be 90%. And so it's a large variety. And we constantly see innovation. We put out an article on Phone Home and Middlebox Reflection, two new techniques that are giving them that ability to conduct large-scale attacks.

Rick Howard: Well, it might help, Jim, if you describe how a customer of Akamai fits into this, right? What puts you in a position to be able to see that kind of traffic and make some decisions about mitigating the attack?

Jim Gilbert: That's a great question.

Rick Howard: How does that work?

Jim Gilbert: So DNS is a really interesting use case because -- and some more background about DNS. It's a leverage point. So if you take down the DNS system, you can infect many other systems because every service has an address.

Rick Howard: It's a linchpin.

Steve Winterfeld: Right. So if you take down the address system, then you infect all the services.

Rick Howard: So, Steve, for you, if you burn my phone book, I'm worthless. That's what that means.

Jim Gilbert: That's right. So it's an interesting use case. And so what a lot of customers do to us, with us, is expand their surface area. So they'll have what are called primary name servers, which are the source of truth for the phone book. And then there are external name servers, which are the public-facing systems that answer questions. And so the source of truth will get often transferred or sent in with an API out to our systems. And then the enterprise will lock down the primary servers and then expose the Internet to our public-facing systems. And so with that, we see a variety of traffic. Given the scale of Akamai, we actually know a lot about the types of traffic that are coming into our systems. And we know we apply reputation. We know the good ones, and the rest are suspect. So the good ones are known resolvers, such as the Google public DNS resolver. We know that company. We work with that company. We know those are good systems. So we assign a certain level of trust to that incoming traffic. Whereas another, what are called recursive resolvers, that is unknown to Akamai, will be suspect traffic. And so when we apply those reputation by scoring, we can actually prioritize the packets. So during times of need, we can give relative priority to good traffic versus bad traffic or suspect traffic.

Rick Howard: Before we started recording here, I used to work at a company called Verisign, and they had a DDoS server. This is back when DDoS attacks weren't as volumetric. Is that the right way to say it?

Jim Gilbert: Yeah, that's right.

Rick Howard: There wasn't as much of that, right? So the customer complaints we got were, you know, I need you to decide that it's an attack sooner, or I need you to not decide it wasn't an attack, it was a false positive.

Jim Gilbert: Yeah.

Rick Howard: Is it easier to detect now because the volumes are so high? Does that make it easier for you guys to do that?

Jim Gilbert: I don't think it's -- certainly the volumes will trigger the limits faster, but the worst thing you can do is block or drop or impact good traffic, right, because then that's an availability issue. So you need systems that are really smart about the incoming traffic. So, for example, if we're getting traffic from claims to be Google resolvers, then we know the IP addresses that they should be using, and we know the networking hop counts out to those systems. So if someone is spoofing a Google resolver IP and it has a strange abnormal hop count, we know the request is suspect. So there's all those techniques that are live and dynamic in zero second. It's algorithmic. There's no playbook or anything. This is just the way the systems need to work.

Rick Howard: There's no person looking at the traffic going, oh, this looks bad. Let me fix it. It's a series of, I don't know, telemetry that you are gathering, you guys, right, and then deciding that --

Steve Winterfeld: Heuristics and telemetry, correct.

Rick Howard: Yeah, okay. That makes a lot of sense, right.

Steve Winterfeld: By the way, Jim, I was very impressed you were able to make yourself say that was a great question. Very impressive. Way to suck up to Rick there.

Rick Howard: And we're definitely keeping that in.

Steve Winterfeld: And so as you think about the different threat actors, they use different techniques and different business models. And the other part is it's been around a long time, but the motivations have changed historically.

Rick Howard: You know, in the old days when you and I were doing this, it was just denial of service, and it would be some piece of virus that would occupy the end point, you know, the victim machine. And then as it progressed, distributed denial of service was network attacks and all the ways you just described. And let's not forget about non-malicious distributed denial of service events. You self-inflicted DDoS on your own systems or your customers did. I was talking to Rick Doten about this this week. He's the CISO for Healthcare Enterprises at Centene, a Fortune 500 company, and he told me this horror story.

Rick Doten: So my non-malicious antivirus story is from about 15 years ago. We had a large hotel chain who had contacted us because their network had come to a screeching halt. They had a whole bunch of traffic. They thought it was some worm variant that they'd never heard of. They had their network folks and some vendors on site, hardware vendors, to be able to, you know, identify what it is, and they couldn't figure out what happened. So I ran the ethical hacking team that doubled as the incident response team when the need arise. I sent some folks out there. They were looking at the traffic and saw that it was virus signature updates, and they traced it back to one server, which was the on-site antivirus server that was supposed to feed all the antivirus updates to all of the systems, the hundreds of thousands of systems around the world. Well, he saw that it was down for 28 days, and it just came back up and had 28 days of antivirus updates pushed out all across the globe, including to, you know, across very small DSL lines to little islands in the Caribbean. So it's not always a malicious act.

Steve Winterfeld: When I was at Nordstrom and marketing sent out a note on a sale, we got overwhelmed. Or, you know, when hospitals announced a website for vaccine appointments, they would get overwhelmed. And so volume is also something we deal with that can be good as well as malicious. So when you're thinking about DDoS, I always encourage people when they look at their playbook, both how are you going to deal with that flood of good as well as a flood of bad?

Rick Howard: Are there particular victims that are more susceptible to this? Like, I'm thinking finance and health care, because if their systems go off and millions of dollars of money is going down the tube or people might die. Is there anybody else that gets targeted for this kind of thing?

Jim Gilbert: So we work with large enterprises, so that's what everyone means to me. And so all the large enterprises of the world are seeing this traffic. And some more than others, I think the ones that see more are the ones that have more at stake. So either people's lives or supply chains or financial systems, those seem to be more likely targets.

Steve Winterfeld: We'll also see campaigns. And it will be interesting, as we have customers in multiple industries, to watch a DDoS campaign walk through industries. They're thinking about who's willing to pay the extortion. So people that are motivated, that are in critical services, exactly, need to give people access to their wealth, their health. Those kind of critical providers, I think those are probably more likely to pay. The flip side of that is you're seeing more and more regulations about paying cyber extortion. So you're seeing both sides of that develop over time.

Rick Howard: So is it fair to say that the current state of DDoS protection is what we've just described here, is a collection of telemetry and automation heuristics to identify the attack in progress and then to mitigate it for your customers? Is that fair to say?

Jim Gilbert: I think that's correct. And I would add one more thing, and that's the network design and scale. So these systems need massive capacity, and they need wide coverage over the world. And these are, again, for the larger enterprises of the world.

Rick Howard: And when you're saying these systems, you're talking about these protective systems from this kind of attack. Is that correct?

Jim Gilbert: That is correct. And there's certain types of attack that you have to let the leaking come through. And the only way to really address the leaking, because, again, one of the worst things you can do is drop valid packets. And so to manage the leaking, you have to actually absorb the attack. And that comes through network design and really wide deployments.

Rick Howard: So the way that a company like Akamai will mitigate the attack is identify the bad traffic and siphon it off into bit zero somehow?

Jim Gilbert: On the DNS side, we'll prioritize it. So we'll get to it when we can. If we know it's good, it's fast-laned. If it's suspect, it's prioritized and we'll get to it when we can. And if it happens to time out, that's okay. If it's a valid client, they'll retry.

Rick Howard: So what I'm hearing you all say is that we have the technology to mitigate DDoS attacks. What do you think is coming in the future? What are you planning for that you have to build now to get ready for that?

Steve Winterfeld: One, as we think about the future, if we're protecting your edge, again, going to those multiple, are we protecting your remote workers, your infrastructure? That's pretty straightforward. We're expecting faster, bigger, more complex attacks. If it's attacking APIs, that's different infrastructure. We're seeing more and more innovation on the criminal side in those environments.

Jim Gilbert: Right. The one thing we've got our eye on is an area called subdomain attacks.

Rick Howard: Yeah. I was going to ask you about that. What does that mean?

Jim Gilbert: In the white paper, you can read details about it. What it means is that there are what are called NX domains. It's garbage traffic. That's our subdomains of valid domains. So it might be asdf.akamai.com. So the DNS systems actually, when they see akamai.com as a valid parent, you need to try to answer it. And so the only way to try to answer it is to ask the source of truth whether asdf.akamai.com exists or doesn't exist. And that consumes resources. Those are called NX domains. And so what we see is very wide botnets that are spreading the world and sending these NX domains to a wide footprint of machines. And then these wide footprint of machines might need to figure out how to answer. And sometimes they'll go forward to the original primary server. Sometimes they'll have the source of truth and just say the answer doesn't exist.

Rick Howard: So these NX domains, these are not something bad guys come up with. They are kind of aliases for other domains inside your own organization?

Jim Gilbert: They can be. There's two different types. There's they're trying to guess at resources like VPNs or anything that might be useful for their abuse. And then there's also garbage traffic, and they may be trying to flood the links. They may be trying to take down the DNS systems. Maybe doing a decoy like we talked about earlier to overwhelm the InfoSec team. But this is a real challenge because, like I said, the worst thing you do is drop a valid DNS request. So you need to we need to look at all these NX domains and answer them. And so if they're coming in wider and more frequent and with bigger spikes for longer periods of time, that'll be something that we'll need to continually invest towards over the coming years.

Rick Howard: So that's something coming down the pipe that will change the environment, I guess.

Jim Gilbert: Yeah, I don't see that stopping.

Rick Howard: Yeah.

Jim Gilbert: And the other really important control for that is provisioning enough capacity, not only distributed. I'll just give an extreme example. So you don't want four machines in Japan and then a couple in Hong Kong. I mean, the scale has to be thousands of machines in these important locations.

Rick Howard: So we're at the end of this. Last word time. Steve, I'll go to you first. What's the thing that you'd like to tell CISOs that they should be thinking about in terms of DDoS protection?

Steve Winterfeld: So I think the major thing is, as you look at what the largest volumes of attacks these peaks are, go back to your current capabilities and say, are you okay? If you get hit by that volume, is your risk portfolio still where you think it is? And then make that an annual review and then always update your playbook. Most people survive DDoS through a third-party expertise. And then, you know, is your playbook up to them? Are you being notified? Do they know who to notify? Avoid the crisis and make it a small incident by making sure that playbook's good.

Rick Howard: So, Steve, you and I collaborated on a book. This is a resilient strategy, right? That's what this is. We need to be able to survive the attack. This is not a prevention kind of a thing. It's a survivability kind of thing.

Steve Winterfeld: Yeah. And when we talk about resiliency or business continuity, this is a highly contested environment. And DDoS is a perfect example of highly contested, an active threat, a thinking threat. And what is your resiliency? And as Jim said, you're not going to do a self-denial service. You're still going to be able to perform your job in the middle of a DDoS.

Rick Howard: So, Jim, last word for you. What should CISOs and senior executives know about denial of service attacks against the DNS? What should they be thinking?

Jim Gilbert: Yes. Often we see that DNS is something that's forgotten. So it's a piece of infrastructure that can be really unique and complex. But just don't forget the DNS, because it is a common target that seems to be targeted more and more. And it has high leverage. So it's something to really fortify.

Steve Winterfeld: And probably not a very funny joke, but we do a lot of emergency onboarding lately.

Jim Gilbert: We have gotten good at it.

Rick Howard: It is funny after the fact, I guess. Well, I have this theory, Jim, that there's only like 10 people that understand DNS. I think I've just met the 11th one.

Jim Gilbert: There's only a few of us. It is an insight. That is a good one. I now know Rick really did work at VeriSign.

Rick Howard: According to Microsoft's Azure network security team, they are seeing roughly 1,500 DDoS attacks daily. That seems like a lot. And yet their customers don't really notice. Most are protected by the sheer scale of the Microsoft cloud infrastructure. That scale enables Microsoft to absorb most of the attacks. And that's true for the other big cloud providers, too, like Amazon, Google, and Akamai. In the general case, the size of these large cloud provider networks allows their respective security engineers to reduce the risk of the hackers' three primary DDoS strategies, volumetric, protocol, and application. And by the way, that was one of the main reasons we all went to the cloud in the first place, to benefit from their large-scale operations, something that I couldn't afford to build and maintain in my little startup here at N2K. But for those organizations that need more protection because they are typical targets, like Fortune 500 companies, because, you know, that's where the money is. Or healthcare providers, because if there is a delay in some of their systems, patients might die. Or content delivery providers, where the business lives or dies based on how available their content is. Then each of the big cloud providers has their own version of DDoS protection services that you can sign up for. It adds additional protection in terms of DDoS telemetry collection and specific heuristic algorithms that Steve and Jim were talking about in this interview. In terms of cybersecurity first principles, DDoS protection is a tactic that falls underneath the resilience strategy. Deploying DDoS protection tools or services is not a preventative measure like zero trust or intrusion kill chain prevention. Instead, they enable an organization to survive a DDoS attack, to continue delivering its services to its customers despite the attack.

Steve Winterfeld: Wow, Rick, you nailed that.

Rick Howard: I'd like to thank Steve Winterfeld, Akamai's Field CSO, Jim Gilbert, Akamai's Director of Product Management, and Rick Doten, the CISO for Healthcare Enterprises at Centene, for helping us get our heads around this distributed denial of service attack pattern. And that's a wrap. Don't forget you can buy copies of my new book, "Cybersecurity First Principles: A rReboot of Strategy and Tactics." Order it now at Amazon or wherever you buy your books. The audio version just dropped so you can now read the hard copy version, read the digital version with your Kindle, or listen to the audible version. And finally, we'd love to know what you think of this podcast. Send email to cyberwire@n2k.com. Your feedback helps us ensure we're delivering the information and insights that help keep you a step ahead in this rapidly changing world of cybersecurity. We're privileged that N2K and podcasts like "CSO Perspectives" are part of the daily intelligence routine of many of the most influential leaders and operators in the public and private sector, as well as the critical security teams supporting the Fortune 500 and many of the world's preeminent intelligence and law enforcement agencies. N2K Strategic Workforce Intelligence optimizes the value of your biggest investment, people. We make you smarter about your team while making your team smarter. Learn more at n2k.com. N2K Cyber's "CSO Perspectives" is edited by John Petrick and executive produced by Peter Kilpe. Our producers are Liz Urban and senior producer Jennifer Eiben. Our theme song is by Blue Dot Sessions, remixed by the insanely talented Elliott Pelzman, who also does the show's mixing, sound design and original score. And I'm Rick Howard. Thanks for listening.