A firewall wake up call.

Dave Bittner: Hello, everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems, and protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Jon Williams: We're diving into this because we're interested in getting more familiar with the SonicWall next-generation firewall platform SonicOS. It's been the subject of a lot of vulnerabilities in the past, and we wanted to be prepared to be able to research in the platform, you know, whenever anything new came out.

Dave Bittner: That's Jon Williams, senior security engineer at Bishop Fox. The research we're discussing today is titled "SonicWall Firewalls are Publicly Exploitable."

Jon Williams: And so in order to do that I went back and looked through the history of all the different advisories that had been released, looking for things that were unauthenticated, had a potential for remote code execution, and didn't have a known proof of concept. The one bug that jumped out at me was this one from 2022 which met all those criteria. So I figured it was a good research target and dove in to see if we could exploit it, and we were pretty successful in the end.

Dave Bittner: Now, looking through your research here, I guess there was some stuff from the folks over at watchTowr Labs that originally caught your eye?

Jon Williams: Mm-hmm. Yeah, just a couple of months back, watchTowr put out a blog post detailing how they found a whole raft of new vulnerabilities in it. They all required authentication, which was less interesting to us and, you know, so we wanted to see what we could find that might not require authentication.

Dave Bittner: Well, let's walk through this together. I mean, how do you get your start when you're digging into something like this and what's the process like?

Jon Williams: Well, in a case like this where we have very little to go on in terms of understanding the nature of the bug, SonicWall's advisory was pretty terse, you know? We knew that it affected the web management interface, but didn't have much background knowledge beyond that. So the typical approach for something like that is to start with patch diffing, right? We acquire software before and after the patch. We extract the files that are relevant for the bug. We analyze them using Ghidra or some other engine like that to decompile the code. And then we compare the code and see if we can identify what has changed, and of those changed functions, which one is relevant to the bug we're trying to exploit. And if we're successful at finding that, then it's a reverse engineering effort to understand what the code is doing, how to reach the vulnerable code, and then, ultimately, how to trigger the bug. And our end goal in this case was to get the crash; although, the bug certainly caught our attention because it has a potential for remote code execution, as well.

Dave Bittner: And indeed, you did find the difference in the code here. What did that difference reveal?

Jon Williams: Well, it was pretty interesting because there was a code pattern which was -- essentially, they were using a snprintf check function, which takes a string and copies it into a buffer, and then it returns a value, you know, which is usually the number of characters that were written into the buffer. Not always, though, and that was kind of at the heart of it. But the issue was that they were -- the developers were using this function two times in sequence, and they were passing the return value from the first call into a parameter for the second call. And so this ended up creating the possibility to pass in a string to the first call that would result in an unexpected output, which would then cause a buffer overflow the second time it was called. And what was fascinating about it was that the snprintf check function is designed to be buffer overflow safe. So if you pass it in a string that is bigger than the buffer that you're trying to write it to, it won't do it. But the thing is, it will copy what it can, and then it will return the length of the string that it was asked to copy. And that was key. Not the value it actually copied, but the length of the string you gave it, even if it didn't copy the whole thing. And the developers, in this case, assumed incorrectly that they would only get back the length of the string that was copied. So when they passed that to the second function, the second function uses it in such a way that it requests a much longer string to be copied than they were expecting. And it ends up overwriting the buffer anyway because they've used it in a way that was inappropriate. So even though the function itself protects against an overflow, they used it in a way that was insecure.

Dave Bittner: Well, help me understand here. I mean, is that -- is that function behaving as designed or not?

Jon Williams: It is, yes, it's just a lack of clear understanding about how it's supposed to be used. And what we're realizing -- or one of the things we learned from this research is that it's a very common mistake. And it's easy to understand why, right? I mean, you'd expect if you're copying a string and it tells you it'll give you the length of the string copied that that's what you'll get. You have to read the fine print on the function definition to understand that there are edge cases where that may not be true. A pretty common mistake and, you know, we're going to be looking for this type of vulnerable code pattern in other places as we go forward because it does seem like something developers do often.

Dave Bittner: Yeah. That's a really interesting insight. So -- so you discover what is at the root of this. What do you do next?

Jon Williams: Well, in this case, we did two things. One, we wrote an exploit for it and confirmed that it worked, so we were able to crash our test target. We, of course, checked the patched version, too, and confirmed that it didn't work, you know, against that. So the developers did patch it effectively. But then once we really understood the nature of the bug, we also looked elsewhere in the code to see if they had made the same mistake in other places. And it turns out that they had. Now, the way that this particular bug is exploited is through an HTTP request that uses a URI path that is way too long. And the original bug, the 2022 bug, had three different paths where you could send that request and crash the server, and they were all fixed in the same place. But what we found by looking elsewhere in the code is that there were actually two other paths where the same bug could, you know, trigger the crash, and those two paths were not fixed in the 2022 patch. So we got really excited at that point and thought we had found a zero-day. But as we continued digging into it, we realized that if we were testing against one of the most recent firmware versions, those paths weren't exploitable. And so we were, like, oh, did somebody already catch this and fix it? Was it unreported? But no, it turns out it was reported in 2023. So a year after the initial patch, SSD Labs found and reported the vulnerability on the other two paths. And they had published their exploit, you know, but we had overlooked it because we were specifically looking for things that didn't have proof of concepts available. Once we had figured out that these two vulnerabilities were, in essence, the same bug, we were able to realize that, you know, the exploit was already out there, but nobody had publicized the paths from the '22 -- 2022 bug that had been exploitable all this time. So we were able to establish a link between the two vulnerabilities, publish our exploit for both of them, and then use what we had learned about testing it to release a tool that could be used to identify both bugs without crashing the target server.

Dave Bittner: I want to highlight that point here. As I was reading through your research, I actually laughed out loud. I'm going to quote it here. It says, "Being able to crash a target is all well and good, but what about identifying vulnerable devices without knocking them offline?"

Jon Williams: Right. Well, that's the thing. I mean, just that it's great to be able to exploit something, but if you have a customer base who's relying on you to help secure their infrastructure, you need to be able to give them proof that they're vulnerable without causing an impact yourself.

Dave Bittner: Right.

Jon Williams: And so, you know, having dug into the bug as deep as we had, we found a way where we could actually exploit part of it. Just enough to know whether or not it was hitting the vulnerable code, but not actually go so far as to cause the crash.

Dave Bittner: And take us through that. I mean, what -- what went into developing that?

Jon Williams: It was a long time of staring at the code trying to understand what on earth was happening. But what it came down to was that the two snprintf check functions that were being called essentially had the end result of copying the HTTP request string into a buffer in two parts, right? So with any HTTP request, there are three pieces. There's the HTTP request method, which is, like, GET, POST, PUT, DELETE, you know? The second part of it is the URI path, which starts with a slash. It tells you where you're sending the request to, and then it may also add query parameters. And then the third part of that string is an HTTP version, which is 1.0, 1.1, 2.0, you know, it's really short. These two functions were copying the second two pieces of that request. So the first function copied the request path into the buffer and then the second part copied the HTTP version into the same buffer. And this is where it ended up being problematic because the return value from the first function call would be used as an offset into the buffer where they were going to write the second part. And so if you ended up getting a much larger return value than you expected, it would advance that pointer out of your buffer and somewhere else on the stack. So that was the first part of the problem. The second part was that when it copied the HTTP version, it was using the input from the previous function to determine the length -- like, how many characters it should copy. And so again, if you got the wrong version -- or, if you got the wrong length back, it would actually underflow, and it would copy a huge amount from your input, you know, when it should just be a few characters, so that HTTP version. And so the difference there is that by changing the length of the URI path that you're sending in your request, you can trigger the first part of it, you know, which is getting a return value that's too big. And by sending an HTTP version in your request that's too long, then you can copy data into the buffer, you know, when it overflows. But if you only do the first part of that, then it doesn't crash.

Dave Bittner: Yeah.

Jon Williams: But it will trigger the patch protection that they added. And the result of triggering their patch protection is that the request just gets dropped, so you don't get a response from the server at all. So by sending a request that has a URI path that's too long but not an HTTP version that's too long, and then looking at the request, the response you get back from the server, you can tell whether or not it's been patched.

Dave Bittner: And you all set about doing this. I mean, you went out and were -- set out to find out how many devices out there were potentially vulnerable, and you came up with some interesting results.

Jon Williams: Yeah. Well, once we had that safe test and we were confident that we could look -- you know, interrogate different servers without causing any negative impacts, it occurred to us to scan the internet, you know? How bad is this, really? Of course, we notified our customers first, but then we did some research. We went to BinaryEdge to build a target set because these firewalls aren't hard to identify online. They have a server response header that uniquely identifies them, so if they're exposed to the internet, BinaryEdge will pick them up. And then we did some filtering to make sure that our target set was accurate and the targets were actually reachable online. So we had a final set of about -- just over 230,000 of these exposed devices that we were testing. We ran the test against all of them and found out that about 76% of them were vulnerable to one or the other of the two bugs, which was kind of mind-blowing. We expected a lot of them to be vulnerable, but not necessarily that many.

Dave Bittner: Wow. What has SonicWall's response been to your work?

Jon Williams: Well, we gave them a heads-up before publishing our research, but this was a case where we didn't really need to do a full responsible disclosure because both bugs were already published. We weren't really exposing anything that was unknown, all we were doing was establishing a link between these two vulnerabilities. So we did give them a heads-up, but we didn't have to do a full disclosure. And they acknowledged and, you know, just wanted to know some more details about it. But from their end, at least, as far as I'm concerned, there really isn't much more they can do. I mean, they put out patches for these bugs two years and then almost one year ago for the latter one. So really, at this point, it's just on the consumers to upgrade their devices. That's the real problem here. SonicWall has mostly done their due diligence at addressing this already.

Dave Bittner: What happens if you use this procedure to crash a device? Would does the recovery look like?

Jon Williams: Well, typically, when you crash a SonicWall device -- it has a watchdog, it'll just reboot it. And that takes a couple of minutes, so you'll knock it offline briefly. But what I found in my testing was that, at least in its default configuration, if you do this three times within a brief period of time, you know, like, within -- I don't know exactly what, it's 10 or 15 minutes maybe -- it'll reboot into maintenance mode. And so it'll be completely inaccessible until an administrator comes in and resets the device. So that -- that could be potentially catastrophic, you know, depending on what the firewall -- depending on what the firewall's role is. You know, who is deploying it for what purpose? If you have the ability to completely knock it offline, it could cause some real serious interference.

Dave Bittner: So what are your recommendations, then? I mean, what should folks do to best protect themselves here?

Jon Williams: Well, there are two things that they can do to mitigate it. The obvious one is to patch. You know, if you install the upgrade, you're fine, you know, you're protected. But, of course, in an enterprise environment, sometimes there are reasons why that's difficult to do. The other mitigation step, of course, is to remove the web management interface from public access. Honestly, the fact that there are 230,000-plus of these exposed on the internet is a problem in itself. The web management interface should not be publicly accessible; it should be only accessible to administrators either on an internal network or through a VPN or some type of secured connection. The best protection, of course, is to do both things; you know, take it off the public internet and make sure it's patched. And that way, it just won't be exploitable at all. But one or the other is good enough.

Dave Bittner: Does the system default to having that interface accessible to the web?

Jon Williams: Well, so it is a web-based interface, but by default, it only exposes it on the internal network, the LAN.

Dave Bittner: I see.

Jon Williams: Not the WAN. So yeah, it all comes down to how you configure it as an administrator.

Dave Bittner: Our thanks to Jon Williams from Bishop Fox for joining us. The research is titled "SonicWall Firewalls are Publicly Exploitable." We'll have a link in the show notes. The CyberWire Research Saturday podcast is the production of N2K networks. N2K's strategic workforce intelligence optimizes the value of your biggest investment, your people. We make you smarter about your team while making your team smarter. Learn more at n2k.com. This episode was produced by Liz Stokes. Our mixer is Elliott Peltzman. Our executive producers are Jennifer Eiben and Brandon Karpf. Our executive editor is Peter Kilpe. I'm Dave Bittner. Thanks for listening.