An IoT educational exercise reveals a far-reaching vulnerability.

Dave Bittner: Hello everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Jake Valletta: So, initially, we had no drive for this protocol in general, so we really were looking more at just smart cameras from a higher level, and we kind of stumbled upon this doing just some independent research.

Dave Bittner: That's Jake Valletta. He's Director of Professional Services at Mandiant. The research we're discussing today is titled, "Mandiant Discloses Critical Vulnerability Affecting Millions of IoT Devices."

Jake Valletta: One of the individuals who was part of the research team was fairly new to the company and expressed an interest in getting into embedded security, so we thought, hey, let's go out, let's get some cameras as a way to do an educational exercise and let's see what we can find. So, we started actually looking at some of the security of the cameras that we were looking at themselves, and we encountered this protocol in this network while doing that research. So it kind of caught our eye once we started digging in at a technical level and saw this traffic that looked pretty unique. And it was exciting for us because it, you know, doing some quick searches online, it wasn't really turning up any results or any immediate, you know, parsers or things to understand the protocol. So that got us excited to actually look into the protocol.

Jake Valletta: And then from there, it ended up just being, you know, some research on our side over the course of several months to learn more about the protocol and then ultimately led to the disclosure that we published earlier this month – or last month, rather.

Dave Bittner: Now the protocol we're talking about here is from an organization called "ThroughTek" and it's their Kalay network. Can you give us an overview of exactly what's going on with this protocol?

Jake Valletta: Yeah, sure. So, the protocol is designed to be very easy to use for companies that want to develop and sell IoT products. So, you know, one of the examples that we've seen is smart cameras, right? So let's say you're a company who's building a smart camera, and you have a lot of things that you want to focus on – the branding, the aesthetic of the device, maybe the functionality, and the camera on the unit. But one thing that's really important is being able to connect the cameras to a smartphone or maybe a desktop application, and that can be a challenging engineering problem for someone to reinvent. So what the Kalay protocol allows is for you to basically use the network that ThroughTek has built to take that out of the equation for your development purposes. You essentially would use their protocols and use their network, and you don't have to think about the implementation. It just kind of works, right?

Jake Valletta: So, it's very attractive to these companies who want to build cameras, because it's one less thing that they have to worry about. And in our experience, it does work very well from a user standpoint. It's fast, it's snappy, the camera feed looks good. So it's one less thing that these developers have to worry about.

Dave Bittner: So, they provide the developers with an SDK that they build in. And I suppose that the folks who are building this hardware would tend to think that this would be secure, being on as many devices as it is. But you and your colleagues discovered some issues here.

Jake Valletta: Yep, that's correct. And there are, you know, there's a lot of different versions out there in the wild. And I think, you know, there's some older devices would run more earlier iterations of the platform. I think, over time, ThroughTek has added features and security controls onto the platform to just make it more robust. But at the end of the day, it really does depend on the, you know, the OEMs and the people building these these devices to, you know, stay current and build in the latest version of the SDK as new versions come out.

Dave Bittner: Well, walk us through the vulnerability that you discovered here. What exactly is going on?

Jake Valletta: Sure. So, the vulnerability that we disclosed is essentially device impersonation. So through our research, we discovered that basically when these devices need to access the network – you have a camera sitting in a house somewhere or some smart device somewhere – it needs to register itself onto the network so that when I open my phone, I can very easily find that device and then establish the connection using the protocol.

Jake Valletta: And what we found is that the way that the registration works is it's really only requiring a single piece of information to actually register and join the network. And that piece of information, as we highlighted in our blog, is what's called the "UID." It's essentially a randomly generated string that's assigned to a specific device, right? So if you have ten thousand or a hundred thousand devices in your fleet, each one of them is going to have a unique UID, and that's the piece of information that gets you on the network. So with that ID, your device will register on the network and then whenever somebody wants to connect to you, they would find you using that ID. And that's how the protocol is able to work so effectively and so quickly, is it's just looking up on their side where that ID lives and making the connection and facilitating the conversation if needed.

Jake Valletta: And what we found is that, you know, because that's the only piece of information, that makes it dangerous and risky for an attacker to be able to, one, obtain UIDs, and two, then use those IDs to stage a more serious attack. So what we found – and the issue again is device impersonation – is that we can masquerade as a device. So if we have an ID for a victim or just anyone, any camera we want to masquerade as, we can register on the network, and maybe it's just a Python script. But the end result is whenever someone attempts to connect to that ID, instead of connecting to the real device, they're actually going to connect to us.

Jake Valletta: And the impact here is when they connect to us – by "they" I mean the mobile application – will then attempt to authenticate to us, because we are essentially the device, so they will send a username and password for the camera or whatever, and then we can capture those as an attacker. Once we have those creds, we now have all three of the pieces of information – the UID, the username, and the password – to be able to then connect to the real device using the Kalay protocol, right? So we can then connect to the actual camerand then initiate a connection to say, view the video or listen to audio or access some of the functionalities that are available past authentication. So that's really at the heart of it what the issue is here – it's being able to impersonate devices and obtain materials that can be used to access sensitive data.

Dave Bittner: And how do you insert your your spoofing device onto the network in such a way that you supersede the actual device?

Jake Valletta: Basically, the way the protocol is designed is, it's at the core, it's essentially a peer-to-peer network, so it's meant to allow various nodes to connect, most of the time, directly to each other, right? So let's say I'm in a coffee shop and a camera is sitting at a house. When I'd like to connect to that camera, I will reach out to the Kalay network and say, I'm looking to connect to this UID, can you help me? And the Kalay network will make a determination based on the various network topologies that we're both sitting in, and say, OK, you two can both communicate directly. Here's the ports and IPs you need to use if you'd like to communicate directly. So it'll essentially broker that peer-to-peer connection for the parties. Or, if the network topology doesn't allow that for some versions of NAT or something, it will actually act as a relay server to get the data to and from the two parties.

Jake Valletta: So what this means is, if I'm masquerading a device, I'm registering a particular device on the network, the server's only remember or know of one device. So let's say there's a device plugged in and it is beaconing out, you know, maybe every minute or so saying, hey, I'm here, log me in and register me – if I'm an attacker and I come along and I start registering over that device, well, the servers get confused and then they think that I'm the actual device, so that when a user attempts to find the real device because that peer-to-peer nature, it's going to then provide the details to the both parties, which is me and not the actual device sitting in the home. So it's – unfortunately the server is maintaining, you know, essentially only one device in their memory, instead of realizing that there's a new device interjecting itself is what's causing the problem here.

Dave Bittner: Hmm. So I mean, help make sure I understand correctly – I mean, the fundamental issue here is that that UID is really the only thing that tells the network that that device is what it claims to be. There's no secondary method of verification.

Jake Valletta: Yep, that's correct. And in some of the mitigations on the platform – and if you read through the, you know, the blog in the disclosure that we put out, you know, the recommendations here are essentially to upgrade to the latest version of the Kalay Protocol. And what that provides is there's a specific feature that ThroughTek developed and it's called "Authkey." And Authkey actually provides an additional layer of security when attempting to connect to a device, which validates that the connecting party is who it claims to be. So it doesn't actually address the fact that you can spoof, but it does mitigate the risk on the other end of it, when an attacker attempts to use it, they will need a secondary piece of information to weaponize the vulnerability. And that's why we've encouraged everyone to upgrade and use this feature, because it's really the easiest win here to fix this.

Dave Bittner: And will that work with older devices? I mean, is this a matter of more the protocol within the network itself and not the – I'm thinking, if I have an old camera that's been sitting, you know, minding its own business on a shelf somewhere for five or more years, is it going to have to be updated or replaced, or will the will the mitigations you describe work even with an older device?

Jake Valletta: So, it's going to depend on the company that's, you know, developing and selling the product, right? They're going to have to produce a new firmware image, which a lot of these cameras in our experience have, you know, an over-the-air or remote firmware update capability. There's nothing, you know, hardware-specific that would make it so you couldn't do this, in our research. But they would need to build a new firmware image with this, you know, updated version of the software as well as using the Authkey. So there would be some changes that would need to take place for manufacturers of these devices to be able to actually secure themselves with Authkey.

Dave Bittner: Well, you mentioned some of the remediations. Can we go through that specifically? What are you and the team recommending here for folks to take care of this vulnerability?

Jake Valletta: Yeah, we have a whole series of recommendations to try to kind of address the issue at various stages of the attack, right? So, first and foremost, we're really recommending that people – and by "people," in this case, I actually mean the manufacturers of these devices – that they upgrade to the version of the Kalay protocol, which provides the Authkey feature. So that's probably first and foremost – make sure the platform is using the security features that are available to them to reduce the risk of this particular vulnerability.

Jake Valletta: Now, the second thing that we want – and this is maybe more or less for the manufacturer of the devices, but more for the companies that sell them – is, you know, a lot of the way that this system is architected is, you know, your mobile device needs to obtain that UID somehow. You know, I'm Jake Valletta and I have a smart camera, and I'd like to talk to my camera remotely. Well, my mobile app needs to get that UID so that I can then connect to that camera using the Kalay network. So it's going to obtain that UID somehow, usually using a web API.

Jake Valletta: What we found is, you know, there's oftentimes an API on, you know, company website dot com, forward slash, get UID or something. And that's fine, but you need to make sure that that API is developed with the best security you can. There can't be enumeration, there can't be brute-forcing abilities there. You really need to make sure that that API is secured adequately, because you can think of a doomsday scenario, which we kind of played through in our heads is, you know, what if you had a company that manufactures a particular model of device – let's say they have hundreds of thousands of devices out there in deployment. What happens if they have a weakness in their API, which leaks UIDs, and I can somehow obtain all the UIDs for a given model? Well, then due to the nature of this vulnerability, I can potentially intercept any one of those connections by just launching the attack with a given idea. I don't know who they are, maybe, but once I have all of them, you can potentially exploit this en masse to do a lot of bad things there. So that's something that we really recommend too. And we put that on the blog that making sure that API security is really crucial to this.

Jake Valletta: And then the last thing is really more of a hardening practice. So, you know, one thing we noticed is it's not just the network here, right? These are cameras that are IoT devices, and they have a level of IoT security that we've seen consistent with other devices in the world, right? So a lot of these devices are running as, you know, a privileged user for – all processes are running privileged. They're not using the latest kernels for Linux. They lack hardening features, you know, specific bits and platform independent execution. So all these security hardening things which you'll see in, you know, let's say, a modern Linux operating system, not all of these are being used in these smart devices.

Jake Valletta: So, you know, thinking, thinking back to the attack, right? If someone's able to perform this attack, it allows them to connect to your your camera, okay, that's bad. But as I mentioned – it's written in the blog – there's a lot of functionality that you can access after you've logged in, right? So things like, you know, rebooting the device or obtaining metadata from the device – it's all accessible over the Kalay network, and sometimes even remote firmware updates are available once you've authenticated.

Jake Valletta: So, all these hardening features that we recommend on the device are really important for further attacks, right? So if I'm able to, you know, establish a connection to a camera out there, you want it to end at being able to view audio/video. Of course you don't want that, but me being able to do like something trivial, like a buffer overflow, and cause, you know, a remote code execution as the root user – well, now I have a shell in your house using this protocol, and now the attack is much bigger, right? So, there's other things that can be done on the camera side, on, like, the hardware itself and making sure that the software that's installed on there is as modern as possible. So, those three things together are what we're recommending that the manufacturers and the developers of these devices use and do to mitigate this.

Dave Bittner: Now, the UIDs themselves – are they complex enough that they're really resistant to brute forcing?

Jake Valletta: So, they're fairly long. They're actually twenty characters long. And in our experience – and this isn't, you know, the full analysis of it – but we did not see an easy avenue for brute forcing. It's a fairly large key space and they do appear to be random. It wasn't very obvious to us that, you know, the first four bytes are the manufacturer, the second three are the year. They do appear to be random, and it's, you know, the full A through Z and numbers. So it's a fairly large key space. And we did some preliminary analysis and determined it'd be infeasible to collide very reliably. So that was not something we fully explored, but we did notice based on the key space it'd be a challenge for someone to carry out.

Dave Bittner: You know, I'm curious, Jake, you mentioned at the outset that you you sort of took this on partially as a, you know, kind of an educational sort of thing, a "What if." For you and your team, I mean, how often do you head down this path? And I guess when you when you head down a path like this, are you expecting to find vulnerabilities? Is that more likely than not what you're going to find when you start down a journey like this?

Jake Valletta: Yeah, I mean, you know, for us in my team, you know, we are professional penetration testers. So this is, you know, sort of an augment to our role, right? We do this professionally for clients day-in and day-out, and we more often than not do find vulnerabilities in the work that we do. And I would say on these side projects, they oftentimes do yield pretty exciting results. This one, I think more exciting than some of the other recent ones. Sometimes you maybe don't find anything that's as exciting as this, but the goal in our mind is to see what's there and what we can find, whether it's vulnerabilities in, you know, the UDP protocol that it's using to talk to on the network or device vulnerabilities in the software – you know, it is the goal, and I'd say the team is quite successful and there's a lot of passion for us to be successful and it's a good energy, I think.

Dave Bittner: Our thanks to Jake Valletta for joining us. The research is titled, "Mandiant Discloses Critical Vulnerability Affecting Millions of IoT Devices." We'll have a link in the show notes.

Dave Bittner: The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Tre Hester, Elliott Peltzman, Puru Prakash, Justin Sabie, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening. We'll see you back here next week.