UPnProxy infiltrates home routers.

Dave Bittner: [00:00:03] Hello everyone, and welcome to the CyberWire's Research Saturday, presented by the Hewlett Foundation's Cyber Initiative. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, and solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Dave Bittner: [00:00:26] And now, a moment to tell you about our sponsor, the Hewlett Foundation's Cyber Initiative. While government and industry focus on the latest cyber threats, we still need more institutions and individuals who take a longer view. They're the people who are helping to create the norms and policies that will keep us all safe in cyberspace. The Cyber Initiative supports a cyber policy field that offers thoughtful solutions to complex challenges, for the benefit of societies around the world. Learn more at hewlett.org/cyber.

Dave Bittner: [00:01:02] And thanks also to our sponsor, Enveil, whose revolutionary ZeroReveal solution closes the last gap in data security: protecting data in use. It's the industry's first and only scalable commercial solution enabling data to remain encrypted throughout the entire processing lifecycle. Imagine being able to analyze, search, and perform calculations on sensitive data, all without ever decrypting anything. All without the risks of theft or inadvertent exposure. What was once only theoretical is now possible with Enveil. Learn more at enveil.com.

Chad Seaman: [00:01:42] Universal Plug and Play was rolled out by Microsoft, and it was intended to make networking in a consumer environment user-friendly and automatic.

Dave Bittner: [00:01:51] That's Chad Seaman. He's a Senior SIRT Engineer at Akamai. Our conversation centers on their recently published white paper, "UPnProxy: Blackhat Proxies via NAT Injections."

Chad Seaman: [00:02:04] If your Xbox, if you're loading up a game and it needs ports 60 through 61000 with UDP, it may send traffic to you on request that you did not necessarily initiate, your NAT will block that. So, if something is attempting to come into your network, but the NAT has not seen you communicate out on those ports, it will not know that that traffic is destined for you and won't really know what to do with it, so it will just throw it away. So, what NAT was made for was your machine behind your router could interact with UPnP and pop holes to destine that traffic to you at all times. Think of it like automatic port forwarding configurations.

Dave Bittner: [00:02:47] And what does NAT stand for?

Chad Seaman: [00:02:49] Network Address Translation.

Dave Bittner: [00:02:51] All right, so that's the intention. But how did it actually work out?

Chad Seaman: [00:02:55] I mean, it works really well for gaming and consumer networks. It meets its intended goal. The problem is there were several flawed implementations that were introduced into the wild by various vendors and software stacks.

Dave Bittner: [00:03:10] So, take us through, what's the history of the vulnerabilities there?

Chad Seaman: [00:03:15] 2006, a researcher came out and basically said, hey, there's a bunch of these UPnP-enabled devices, whereas it's a LAN scoped technology, it's not meant to ever be on the internet, it's meant to run your local network. Some of these daemons that are running this technology on these devices, it's basically listening on both interfaces. It's not just listening on the LAN side of the network it's listening on the WAN side, and it's responding. And that is the fundamental problem.

Chad Seaman: [00:03:43] There have been other vulnerabilities discovered in the UPnP stack, remote code execution and command injection and stuff like that. But all of this primarily centers around the one flaw, that is, parts of the technology that should never be on the Internet are listening on the interface that faces the Internet and responding.

Dave Bittner: [00:04:02] I see. So, walk us through that, how does this NAT injection exactly work?

Chad Seaman: [00:04:07] The first step is called SSDP, which is Simple Service Discovery Protocol. It's the UDP portion. So, if you have ever been on a network, a local network, and you've opened up Wireshark and started listening, you would start seeing all of these connections and things coming through that are broadcast UDP traffic. And they're basically doing what's an SSDP service discovery requests.

Chad Seaman: [00:04:33] So, anything running on the network that's running SSDP will see these requests come in and respond back and say, hey, I speak this language and here are the services I offer, because, once again, we're assuming we're on a LAN network and the devices there are trusted.

Chad Seaman: [00:04:49] And this is how these devices expose media streaming capabilities and, you know, how a device can find the router that it needs to poke holes in for NAT purposes. So, the first step is that UDP broadcast, which people are already using for DDoS purposes. But the response that comes back actually has some more information inside of it, and some of that information is the location as to where you can find the UPnP/TCP components, the daemons that control NAT and other service offerings for that device. So, for the routers, you can hit them with an SSDP request; if they respond to that, it means that's already one flaw, right?

Dave Bittner: [00:05:29] It shouldn't be responding to the outside world.

Chad Seaman: [00:05:32] Correct. And when it responds back, it tells you where to find the UPnP daemon, but the UPnP daemon is scoped with the IP address assuming that you're on the LAN. So, you know, it might say go to 192[.]168.1.1, port 56538, slash, and then give you a whole path, potentially sometimes even with the UIDs of the device to make it so you can't just guess it. You know, so it was designed to be not really able to be found super easily, or guessable, and you take that and change that IP from the internal to the external IP that you just communicated with to get the SSDP response, and there is a chance that you will be talking to the TCP daemon that is the UPnP functionality.

Chad Seaman: [00:06:23] From there, it leaks all kinds of information. You can find manufacturer, model names, serial numbers, firmware versions, capabilities, service offerings, and then from each of the service offerings you can drill down and find even more capabilities. You can, in some cases, remotely reboot stuff. You can see transfer statistics, you can see all kinds of stuff. You can even subscribe to events. If you would like to know when this device does things, you can potentially subscribe to those events remotely, and have it ping you when something happens.

Chad Seaman: [00:06:59] In this case, the UPnP daemon is exposing the Internet gateway daemon, I believe it is, but it's called the IGD in these services lookups. And this is what allows you to see what UPnP forwards are in place, and allows you to also inject them.

Dave Bittner: [00:07:18] Take us through that injection. How does that work, and then what would people use that for?

Chad Seaman: [00:07:23] People would use it for, in a regular functional safe network, they would use it for, just like we described with the Xbox, popping holes in the NAT for certain UDP ports or something. In this case, you basically have to craft a SOAP request. You can do it in a single curl command. In the white paper, we actually give some examples as to how easy it is to do, and the commands to potentially do it, or at least the ones we used in our research. Well, people right now are using it for bad stuff. They're using it for proxying traffic out of devices they don't own, and we can only assume for what they're doing that for.

Dave Bittner: [00:08:02] Let's walk through some of the things that people are using this vulnerability for. What are the bad guys doing?

Chad Seaman: [00:08:08] Well, we don't know. That's the problem. We don't have any honeypots, or anything like that, currently running to intercept this traffic to see exactly how they're using it. The only thing that we were able to do is kind of passively look at what they have injected into these devices, and then try and draw some kind of conclusion from that. You know, our assumption is that we're seeing it used for, you know, account takeover, potentially credit card fraud, ad click and affiliate fraud campaigns. And all of these things have potential to make a lot of money. So, there is incentive there for them to do that.

Chad Seaman: [00:08:43] The only real case that we know that somebody was using it was actually discovered by Symantec as part of an APT group that were running the Inception framework, is what they were calling it. And the bad guys were basically storing malware on a WebDAV web service for distribution purposes. So, this is just where they stored their malicious files. So, to hide their own tracks, they had chained together multiple UPnP proxies. So, when they were interacting with the web service, it would appear to be some person's home router, you know, some other IP. And the reality is that, before they even jumped through that IP, they jumped through multiple other UPnProxy compromised devices to get there. So, it's kind of like that meme you see on the Internet, you know, "good luck, I'm behind 7 proxies."

Dave Bittner: [00:09:32] Right.

Chad Seaman: [00:09:33] That was literally what they were going for.

Dave Bittner: [00:09:35] Now, if I'm the person whose router has been compromised, would I know that anything was going on?

Chad Seaman: [00:09:41] No. You would have no clue. There's no signs on the device. So, for UPnP in general, like I said, it's an automatic device that's meant to be used by, you know, in an automated fashion from one device communicating to another device. It's not supposed to be a hands-on human experience, and because of that there's no way for humans to easily inspect, audit, or manage it straight out of the box.

Chad Seaman: [00:10:11] If you log into your router and you start digging around and looking at your NAT table, first of all, the only thing that's going to show up for you in your port forwards are the port forwards that you have put there. Anything that was injected into your NAT table as a port forward by the NAT Daemon, or the UPnP daemon, will not show up there. There's not really a good interface for dealing with that. I think there are some tools on Windows that will let you kind of audit a UPnP device. But what we basically opted for was some shell scripts and writing SOAP requests, because that was pretty straightforward.

Dave Bittner: [00:10:46] Yeah. But again, I'm imagining, if you're someone that this has happened to, I guess part of the vulnerability is that someone could be using your router to proxy data through, they could be up to no good, and someone investigating that could end up on your doorstep?

Chad Seaman: [00:11:05] Yeah, and that's exactly the reason that we wanted to publish this research. When we initially found it, it seemed like it was being used by parties to sidestep censorship efforts, to get access to uncensored information. At first, I was really hesitant to throw cold water on that effort. When we started digging deeper into it, it was like, no, no, this is much bigger and much worse. And the potential for abuse, you know, I think what we're seeing now pales in comparison to the potential that this has to introduce to forensic investigations and law enforcement efforts.

Chad Seaman: [00:11:43] So, I mean, think about it like this. I could set up a malware distribution server, and I could go around to, when we did our basic head count there were 750,000 of these things that responded to our TCP probes. So that's 750,000 potential exit points I could find now. All I have to do is inject one port pointing back to my server into what, fifteen, twenty thousand of these machines, compile that list of what the IP was and what the port was, and then write that into my loader distribution software.

Chad Seaman: [00:12:18] So now, an investigator that's being infected by this is going to think that we have twenty thousand C2s, or when they look up at these devices or these IPs that this stuff is coming from, they'll basically have clean reputations. They're all sitting on residential networks, consumer grade stuff, where you would not expect malware to be coming from. And then you start to draw conclusions. Is it a machine behind that thing that's infected? Like, is it the router itself that's infected?

Chad Seaman: [00:12:45] And at the end of the day, none of those answers would be true. If you were investigating a potential infection, and you were to go knock on a door and ask to see a potentially compromised device, that potentially compromised device would not exist on the network, and it would not be on the router itself.

Dave Bittner: [00:13:03] Because it is the router, and yet the router itself doesn't reveal, you can't just get into the interface and have it reveal what's going on under the hood, by design.

Chad Seaman: [00:13:14] Unless you were to explicitly go out of your way to inspect the UPnP NAT tables, you would never find the reason. Nothing weird would seem out of place until you looked at the UPnP table.

Dave Bittner: [00:13:27] And so, it seems like this would be a good avenue for DDoS and botnets as well.

Chad Seaman: [00:13:32] Yeah, it could. I mean, that's the thing, if we talk about the distribution of malware, the same conversation should be had about botnets and C2 communications. If I needed to hide my C2, why not stick it behind seven proxies, right? If you're jumping through multiple residential gateways to communicate with my endpoint, you're going to be thinking it's some kind of P2P network, you're going to be thinking some kind of crazy transient C2, and the reality would be that, no, it's just one C2, it's just tucked behind multiple residential gateways that you don't even know to inspect.

Dave Bittner: [00:14:07] Now, if I wanted to, I don't know, to fix this, if I was afraid that I might be falling victim to this, is this a situation where I could do some kind of hard reset on the router and start from ground zero?

Chad Seaman: [00:14:19] Yeah, you could. You could do a hard reset on the router, and as soon as you did that, you would want to go and disable you UPnP. The problem is that UPnP is a pretty handy functionality for the LAN side of the device. Before you do that, you should inspect, or maybe start Googling around and seeing if your model or manufacturer could be impacted by this, if there's known vulnerabilities in the UPnP stack. We also included, in the white paper, there's a small script with a payload, where you can point it at your own home gateway, and it will attempt to dump the NAT table into a terminal for you, so you can actually inspect and see what's in there.

Chad Seaman: [00:14:58] And the presence of something being in there isn't scary. It's only scary if you're seeing other people's stuff in there. So, you know, you'll see your 192.168s, like, your torrent clients. Skype injects stuff, WhatsApp inject stuffs, like, different apps are injecting stuff into this table for their own personal usage. So, it's not scary. It's just when you start to see Internet routable addresses showing up in there, that's when something is wrong.

Dave Bittner: [00:15:24] Now, in terms of the manufacturers, how would they address this? Is there acknowledgement that there's an issue? Have there been any updates? Do new models that are for sale right now, are they still vulnerable to this?

Chad Seaman: [00:15:37] Well, it's kind of a crapshoot. You can't really tell. The disclosure happened in 2006, which gives us plenty of time to actually fix the problem. And just our quick and dirty head count of device models, manufacturers, based on the information leakage from the UPnProxy, the UPnP payloads that came back, we identified 72 manufacturers and 400 models. Some of those manufacturers came back in dispute, and we were like, okay, well let's dig deeper into this and make sure you're wrong and we're right, or vice versa.

Chad Seaman: [00:16:10] And we actually started digging up more models for some of those manufacturers that we didn't even notice in the first pass. So, there's actually far more devices that are vulnerable than what is listed in the actual device section of the paper, because of time constraints, and we needed to get this paper out. I wish I could say that there was a lot being done about it. Some of the manufacturers have come back and said that those devices are not going vulnerable. They've tested them. But the fact that we were even able to identify them from Internet-based scans would suggest otherwise. I haven't heard a whole lot about them rolling out patches, or notifying users, or anything like that.

Dave Bittner: [00:16:46] From what you've been able to examine, from what you've been able to gather with the research that you've done, how widely spread is this problem? Should I assume that my home router has been compromised, or is it something that's, you know, there are so many routers out there that it's few and far between the ones that have actually been infected?

Chad Seaman: [00:17:05] I mean, there's millions upon millions of home routers out there that don't have this problem, right? I believe in the paper we said that, when we started the scans, it was about 4.8 million that were identified via the SSDP leaks. Of that 4.8 million, 750,000 responded to TCP probes, and of that 750,000, 65,000 were found with injections that pointed to Internet routable devices or IPs.

Chad Seaman: [00:17:32] So, that's not to say that only 65,000 of them are vulnerable. 750,000 of them are potentially vulnerable, 65,000 were actively being exploited. There were others where we were able to communicate with the daemon, but there was nothing in the NAT table that suggested anything suspicious was going on. It looked like just regular 192.168 injections from devices living on the LAN. So, you're talking a small percentage of the overall footprint, but it's still a significant number if you're trying to defend against it, and it's still significant if it's happening to you. You know, if the FBI comes knocking on your door, you're not going to care how many other people it's happened to.

Dave Bittner: [00:18:15] Right, right. You should keep a copy of this white paper printed out and sitting next to your router just in case.

Chad Seaman: [00:18:23] (laughs) Exactly.

Dave Bittner: [00:18:23] All right. Well, I mean it's an interesting story. It's one of those ones that I guess is to be continued, because it's happening, it's hard to defend against. So, I guess it's one of those ones we'll have to see how it plays out. I guess, for you, frustratingly unresolved.

Chad Seaman: [00:18:41] It's frustratingly unresolved, and it's one of those topics where I feel like we need people looking at it, we need people to know it exists, but we also risk stoking the fire of adoption by even mentioning it. You know, I think it's important for forensic investigators and law enforcement stuff to know that this capability exists, and to know to look for it when they're dealing with their investigations. But, from the DDoS perspective, and from the blackhat proxy perspective, from the carding perspective, and the account takeover perspective, it also presents a serious threat that I don't know if we've actually seen it really leveraged yet.

Dave Bittner: [00:19:26] Our thanks to Chad Seaman from Akamai for joining us. The white paper is titled "UPnProxy: Blackhat Proxies via NAT Injections," and you can find it on the Akamai website.

Dave Bittner: [00:19:39] Thanks to the Hewlett Foundation's Cyber Initiative for sponsoring our show. You can learn more about them at hewlett.org/cyber.

Dave Bittner: [00:19:47] And thanks to Enveil for their sponsorship. You can find out how they're closing the last gap in data security at enveil.com.

Dave Bittner: [00:19:55] The CyberWire Research Saturday is proudly produced in Maryland, out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technology. It's produced by Pratt Street Media. The coordinating producer is Jennifer Eiben, editor is John Petrik, technical editor is Chris Russell, executive editor is Peter Kilpe, and I'm Dave Bittner. Thanks for listening.