Research Saturday 7.16.22
Ep 241 | 7.16.22

A record breaking DDoS attack.


Dave Bittner: Hello, everyone, and welcome to the CyberWire's "Research Saturday." I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Chad Seaman: We started seeing this attack showing up in the real world, and the patterns of the attack were interesting enough to start to dig into it more and try and figure out what exactly we were looking at. 

Dave Bittner: That's Chad Seaman. He's a team lead with Akamai's SIRT. The research we're discussing today is titled "CVE-2022-26143: TP240PhoneHome Reflection/Amplification DDoS Attack Vector." 

Dave Bittner: So at the core of what we're talking about today is a reflection/amplification distributed denial of service attack - a DDoS attack. For those who may not be all that familiar with it, can you give us a little brief overview of exactly how a reflection/amplification attack works? 

Chad Seaman: Yeah. So DDoS in general - I often describe it as kind of like the Christmastime rush. So, you know, if you go to a store around Christmastime, they may have 20 different lanes open for you to check out. And, most likely, all of those will be open, and there will probably be a short line in all of them. Now, if you walk in on a random Tuesday in the middle of June, that may not be the case. There may only be one or two registers open. There may not even be any humans standing there scanning products. It may just be the self-checkout lines, right? And those lines will be manageable. A DDoS attack would basically be like a flash mob running into a Walmart the size of a Christmas shopping crowd and then filling up all of the queues without the company having time to bring on enough staff and enough hands to support that level of traffic in the physical world. A DDoS is that in a virtual format, right? 

Chad Seaman: A reflection attack is slightly different, only in that the attacker pretends to be the victim, and then they go out and essentially simulate a request as the victim, which initiates a response from the reflector. This would be kind of like I go out, and I put a - that you're selling gas at 55 cents a gallon. And then I put your phone number on the internet, and your address, and all of a sudden people start showing up and calling you. And now you're just overwhelmed, even though you had no intention of selling gas, or you may not even have gas, but the people calling you don't know that. They - all they know is the information they were given, which is, hey, if I reach out to this person, they'll sell me gas at 55 cents a gallon, so I'm going to get on that. And they're going to do it at scale. 

Dave Bittner: Well, let's dig into this particular effort here. What exactly were you all observing? 

Chad Seaman: So these attacks started to show up, and they - I mean, they weren't devastating attacks, but they were concerning. It's always interesting when we see a new vector pop up. In the landscape of DDoS, especially at the scale of it, several of the other researchers that participated in that paper, and especially Akamai - it gets pretty boring, to be completely honest. You see a lot of the same stuff day in and day out. So when you do start to see a new pattern or a new something pop up on your radar, it's always interesting because it means that somebody's figured something out that you haven't yet discovered. And this was the case. 

Chad Seaman: In this case, you know, because of the way that reflection attacks work, like I said, you have to go forward and pretend to be a victim and talk to a service. Part of that response that that reflector sends back to the real victim once it routes across the internet is the source port of the service that is being abused. So when we start to see new source ports show up in this type of traffic and then within the traffic itself, there are patterns that can be identified as a side effect of the protocol that they could be abusing. Those are all interesting. 

Chad Seaman: So, you know, if somebody is running NTP on a weird port and we see these weird payloads coming in from a weird port - or known payloads, rather, coming in from a weird port, it doesn't raise as many eyebrows. But when you start to see a truly new protocol or truly new content or - pattern in the attack traffic also coming from ports that aren't traditionally leveraged for reflective DDoS, it kind of sounds the alarm. Like, oh, man, OK, so there's something new out there. We don't exactly know how bad it hits, but we know that somebody is already abusing it because we're seeing it leveraged in attacks right now. And then the fact-finding begins. 

Dave Bittner: Yeah. 

Chad Seaman: How is this abused? Why is this abused? Is it misconfiguration? Are there a million of these things? Or are there a thousand of these things? And then, you have to start figuring that out in real time as you start to learn the landscape and figure out how to best fight it. 

Dave Bittner: And this is an interesting one. I mean, walk us through the process of how you and the other folks on this team collaborated to find out exactly what was going on here. 

Chad Seaman: You know, like I've said, it started showing up and looked fairly interesting. From there, using real world attacks, we were able to passively identify some sources that we could talk to because the one thing that reflection attack can't hide is the true source of the reflection node. If we see them talking to us on - I think it was port 10074, we could go back to port 10074 from an attack source that participated in an attack and start to interrogate that service running on that real machine that was leveraged in the attack. So from there, once we start to communicate with that - we had no visibility - right? - into what the command structure or how that service functioned at all. We only had the payloads that were a result of somebody issuing a command to that endpoint. So you kind of go forward and start to poke and prod and figure out, what is this thing? How do I talk to it? Thankfully, for these, because of them essentially being a test interface for different capabilities of this device, if you sent anything - a question mark or a space - it would dump back a help screen. So right there, you basically had self-documentation of all the capabilities of the device. And then, it became a game of, well, which one of these are they abusing? It was pretty obvious which one was being abused based on the payloads that were coming back especially once you got the help screen. And then, you start to kind of delve into that. 

Dave Bittner: And this was a voice-over IP system that was being taken advantage of? 

Chad Seaman: It was a software/virtual device/physical device that kind of predated, like, a Microsoft Teams offering. So it - the device itself could be purchased, and it was mostly purchased by small organizations that predated some of the platforms and stuff that we have now that allowed them kind of a unified hub for file sharing, communications, chat. And as part of that, there was some PBX integration that allowed it to also engage in phone calls - sending and receiving, I believe. 

Dave Bittner: And that's what the folks were able to take advantage of, that those were opened up to the public internet. 

Chad Seaman: Yeah. And early on in the initial phases of this when we kind of started to figure out what these things were, that was one of the big floating questions. You know, we found some of these devices were in very precarious networks. There were some that were linked to towns' security - or not security - emergency response infrastructure. So at that point, we kind of had to start a deeper fact-finding because we weren't sure - when somebody is abusing - and I'm sure we'll go into more as we discuss the research. But these devices are single-threaded. So we found that if one was being leveraged for an attack, any other attempts to communicate with that service failed. They would just hang until that attack was finished. And that led to the question of, like, are people leveraging these to attack us successfully knocking down small towns' 911 infrastructure or emergency response infrastructure? And, you know, those are just some of the questions that pop up early on. And you kind of - those are questions you have to answer. And you need to answer them fast because it really - it changes the overall threat, right? 

Dave Bittner: Yeah. 

Chad Seaman: So if somebody is DDoSing me and it's hurting, that's not good. If somebody is DDoSing me and it's taking down Tacoma, Wash.'s 911 operations, that's a bigger problem. So luckily, we found out that that was not the case. Thank goodness. 

Dave Bittner: I see. Now, this attack had a particularly high - in your research, you say it was a record-setting amplification ratio. What was going on there? 

Chad Seaman: Yeah. So typically, in these reflection attacks, the attacker has to continually engage with the reflection component. So if I'm talking to - and this is in most cases, this is not 100% across the board case. But in most cases, if I'm talking to Server A and I'm reflecting traffic at Victim Z, I need to send a packet to Server A, which in turn triggers a response to Victim Z. In this case, because of this being a test facility that was exposed and leveraged UDP, you could basically send a single request, and that request would be processed by that test facility. That would result in millions of packets exiting the reflector node targeting the victim. 

Dave Bittner: Was this the result of a configuration error? Or was this something, you know, overlooked in how the system was designed? This is an older system, yes? 

Chad Seaman: Yeah. You know, when I first started testing it - so pretty early on once we started to identify what the device was, reaching out to the manufacturer and trying to get the ball rolling on cleaning stuff up, like, I said, there was a - there is a virtual device component of this product. So we were able to actually get copies of the OVA images and deploy virtual devices. And in my initial deployment of a virtual device, it wasn't vulnerable. And what I found was that during the setup phase, it asks for a public interface and a private interface, and then it exposes certain things depending on that. And if a user, during the configuration phase, had configured it to use the public IP address as both the public and private interfaces, then this misconfiguration occurred. And basically, this testing card thought, well, this is my private internal interface, so I can expose everything to it. Or so is my relatively hot take. I haven't really dug super deep into exactly how they get into this configurable state, but in my early testing, that was something I discovered because I'm like, great. I've got this virtual device. Let me just start beating on it, and it's not working. So yeah. 

Dave Bittner: Yeah, that's fascinating. Well, where do we stand in terms of mitigation? Do, you know, standard DDoS defense tools work against this? 

Chad Seaman: Yeah. I mean, if you're engaged in, you know, upstream filtering and that kind of stuff to shed traffic, then you're going to be okay. The problem is it's a volumetric attack. So if it's possible to overwhelm your total bandwidth anywhere in the upstream, you're going to be hurting regardless. And your mitigation options are rather limited if they can push enough traffic. All the standard plays that you would use - like I've said, this is - DDoS is fairly boring because you see the same thing over and over and over again. And a new reflection attack is interesting in that it's new, but it's not interesting in that it's all that unique. The unique part about this one was the one attack or packet can result in 220 billion amplification rate. But at the end of the day, if you were to still do source port blocking of traffic and ACLs and a lot of the standard playbooks, you would be okay so long as you had enough bandwidth to support up to your mitigation gear. 

Dave Bittner: Now, you've been in touch with the manufacturer of this device that's being taken advantage of here. What's their response been? 

Chad Seaman: Yeah. So it's funny. We reached out to them, and at first they were aware of the situation in the initial outreach. And then as we further - that discussion went down that rabbit hole, it became clear that while they were - had been made aware of the problem, they weren't fully aware of the severity of the problem. Like I said, in the initial interaction with the device, if you send anything, even a single byte you get back, I think like 64 bytes or 128 bytes of response that is that help message. It's basically an error message followed by the help message, which - you know, yeah, that's a reflection and amplification concern. But they weren't aware that using some of these other test interfaces that that service exposed really pushed it up to record-setting levels of potential. Once that was more thoroughly explained, they were very proactive and cooperative in going through the process of trying to identify their customers and contact them. And it was tricky because they create the devices. And it's - on top of that, it's a legacy device that I don't think they really sell anymore and don't even support, really. But they also have partnerships, and those partnerships were also partially responsible for the selling, configuration, deployment and management of some of these devices. So it became kind of a game of whack-a-mole of trying to track down all the devices that we could identify on the network - on the internet, sorry - which network that those devices lived on and then whose responsibility or what contractual agreement they have on the back side of their business that will ultimately get them in a position to be able to talk to the customer that has deployed that device. 

Chad Seaman: So it was a little bit tricky. We had actually planned to go public with our findings earlier, but once we got involved with the contacts at Mitel, basically, we stretched out that timeline to provide them with more time and opportunity to clean up and try and get customers notified and patches put in place and firewalls spun up and everything that - you know, that comes with responsible disclosure. 

Dave Bittner: It really does seem like, you know, one of those tricky sorts of things, particularly when you're working with legacy equipment like this. I would imagine there are a lot of organizations who have things kind of running behind the scenes, and they probably think to themselves, well, we're not sure how much this is being used, but we might as well leave it running because it's not really going to hurt anything. 

Chad Seaman: Yeah, yeah. And it's funny because while we spotted the abuse because of it being leveraged for attack, that's also, ultimately, how Mitel was made aware of it initially - was because companies had these things deployed, and who knows how long they've been there or if they're still actively being used. But when somebody is abusing them for DDoS, it shows up in your outbound traffic. So suddenly, you start seeing this device on your network, and it's consuming massive amounts of outbound traffic. And that ultimately was what kind of raised up one of their customer's ears that notified them of this problem. And then as those attacks became more prevalent and we started seeing them show up across our borders, targeting our customers and such, it became a case of, oh, OK, well, you know this is there, and your customers are telling you it's abuse - and you - they were proactively trying to get it fixed before we even reached out to them - but they just didn't understand the severity of it. So it was definitely tricky. 

Dave Bittner: One of the things that strikes me about this is the degree to which the research and the mitigation, as you mentioned at the outset, was - is this really a task force? There are a lot of - it's practically a who's who of research teams who collaborated on this. Can you touch on the importance of that - of sharing this sort of information across the industry? 

Chad Seaman: Yeah. So within the industry, there are some smaller groups that - that kind of try to collaborate. And I've discussed this in some interviews in the past and such about some of these working groups that have spun up. And it's because DDoS is, you know, it's - it's not just an Akamai problem - right? - it's an everyone problem. And in some cases, it kind of also needs to be an everyone solution. We can sit back and mitigate some of these attacks - all of these attacks - up to this point without any problem or any concerns, but you might not be able to, and somebody else might not be able to. It really does become a bigger question of, well, we should probably fix this for all of us. And that's kind of the driving force behind some of these collaborative efforts. So while we are direct competitors across a particular industry, we're also interested in making sure the internet stays up and functional and healthy. And, you know, no single group or bad actor can come around and start really causing widespread pain for everyone. 

Dave Bittner: Our thanks to Chad Seaman from Akamai for joining us. The research is titled "TP240PhoneHome Reflection/Amplification DDoS Attack Vector." We'll have a link in the show notes. 

Dave Bittner: The CyberWire podcast is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Rachel Gelfand, Liz Ervin, Elliott Peltzman, Tre Hester, Brandon Karpf, Eliana White, Puru Prakash, Justin Sabie, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening. We'll see you back here next week.