The security implications of cloud infrastructure in IoT.

Dave Bittner: [00:00:03] Hello everyone, and welcome to the CyberWire's Research Saturday, presented by Juniper Networks. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Dave Bittner: [00:00:25] And now a quick word about our sponsor, Juniper Networks. NSS Labs gave Juniper its highest rating of "Recommended" in its 2019 Data Center Security Gateway Test. To get your copy of the NSS Labs report, visit juniper.net/SecureDC, or connect with Juniper on Twitter or Facebook. That's juniper.net/SecureDC. And we thank Juniper for making it possible to bring you Research Saturday.

Dave Bittner: [00:00:55] Thanks also to our sponsor, Enveil, whose revolutionary ZeroReveal solution closes the last gap in data security: protecting data in use. It's the industry's first and only scalable commercial solution enabling data to remain encrypted throughout the entire processing lifecycle. Imagine being able to analyze, search, and perform calculations on sensitive data, all without ever decrypting anything – all without the risks of theft or inadvertent exposure. What was once only theoretical is now possible with Enveil. Learn more at enveil.com.

Matt Chiodi: [00:01:34] So, one of the things that we've really observed over the last, let's say, a year-and-a-half is just this great shift in how organizations are building their cloud infrastructure.

Dave Bittner: [00:01:46] That's Matt Chiodi. He's Chief Security Officer for Public Cloud at Palo Alto Networks. Matt Chiodi is going to be discussing the Cloud Threat Report from Unit 42. Later in the show, he'll be joined by Ryan Olson. He's VP of Threat Intelligence at Palo Alto Networks, and he leads Unit 42. He's going to be discussing the Unit 42 IoT Threat report. Here's Matt Chiodi on the Unit 42 Cloud Threat Report.

Matt Chiodi: [00:02:11] So, let me give an example. So, in the past, if I was a developer or if I was someone in IT and I needed to go and provision cloud infrastructure, right? So think of, you know, AWS, Azure, Google – in any one of those environments, in the past, I would have to go and manually create things, right? So I'd have to log into a cloud console. I'd need to, you know, click here to create a new server. Click here to create networking. Click here to create storage. Well, what's radically changed is this move toward infrastructure-as-code, or IaC, infrastructure-as-code templates.

Matt Chiodi: [00:02:47] And it sounds fancy, but really all infrastructure-as-code templates do is they create the basic building blocks for how cloud infrastructure is largely now created. And that's a good thing. But what we found was, you know, we wanted to look at what are the security implications of moving towards this infrastructure-as-code? And again, all that means is that instead of me going out and manually creating cloud infrastructure, I now design it on a whiteboard, I put it into code, and I can now re-use that template as many times as I want.

Matt Chiodi: [00:03:22] Now, the security implication comes here – is that what we've known from both past research and also from this most recent report is that poor cloud security practices are rampant. One of the headlines that we kind of found as we sifted through just, you know, petabytes of data, is that we found over two hundred thousand insecure templates in use. So, again, these are those are infrastructure-as-code templates. Some people might recognize the term – like, for example, AWS has its own CloudFormation Templates. HashiCorp makes – it's something called Terraform. Azure has ARM templates, right? These are – each one has their own proprietary template.

Matt Chiodi: [00:04:02] What we found specifically was our research team went out and we did the first ever large-scale study of these templates. So, what we did was we went out to GitHub, we used their searching API, and we literally downloaded hundreds of thousands of these templates. And then we ran it through our Prisma Cloud API scanner, which looks at templates for common misconfigurations and vulnerabilities. And this is where we really started to get the data that we'll talk about here over the next couple of minutes.

Dave Bittner: [00:04:33] Now, when you say that these templates come with insecurities, I suppose there's a spectrum of seriousness to those issues, yes?

Matt Chiodi: [00:04:44] Absolutely. And so, not everything that we found was, you know, a massive criticality. But in that two hundred thousand number, each of those templates had at least one or more medium or high-severity vulnerability. So an example, right, of a high-severity vulnerability, what we would consider a high vulnerability – a high-severity vulnerability – would be, for example, if a template exposed a database to the public internet. That's an example of a template creating a high-severity vulnerability. Another example could be a infrastructure-as-code template that exposes an S3 bucket to the public internet, right? And of course, there's pieces of it that also come into that as well. But those are just some kind of very high-level examples of what we would consider a high or maybe even a medium-severity vulnerability. Of course, it depends upon the type of data that's also behind that, right? But from just analyzing just this massive number of templates, which has never been done before in the industry, we were able to kind of pull some of these statistics out.

Matt Chiodi: [00:05:50] So, let me give you another example of what we saw in there. We found that – again, by analyzing this massive number of infrastructure-as-code templates, we found that about 43 percent of cloud databases are not encrypted. This is important, right? Because unencrypted cloud databases can lead to data breaches, right? And MoviePass is a recent example of that. So, there's just things that obviously, even with, you know, not having databases encrypted, depending upon the nature of the data, there could be compliance issues there as well, right? Things like PCI and HIPAA all require encryption of that type of sensitive data.

Dave Bittner: [00:06:31] Yeah. One of the other key findings that you highlight here is that 60 percent of cloud storage services – they have logging disabled. That really struck me. What's behind that number?

Matt Chiodi: [00:06:44] Well, obviously, I don't believe it's intentional, right? I think this is one of those things where a lot of times you have developers that are creating these code templates, right? And they're creating it for development environments, they're trying to make sure things work. And when you do that in development, a lot of times, you know, you turn certain things off, right? Logging, although the costs in the cloud is is very, very low compared to what it used to be on-premise, there's still a cost with it. And so, you know, one of our things that, you know, we've kind of taken away from this is that probably it was left that way from development. And when it was then carried over into production, there was then no scrutiny as to, hey, you know, we need to have logging enabled, right? Because obviously, when cloud logging is disabled, attackers could enter a cloud storage system and an organization would never know.

Matt Chiodi: [00:07:33] So, think about if there was some type of data breach. If that would happen to an organization and they don't have cloud storage logging turned on, how would they be able to disprove or prove that there was or was not a breach? It's really important, if not even impossible, to do any type of what we would call attribution if you don't have a record, right? And this is – you know, if you think about, you know, kind of a – you know, a less digital example would be, you know, if you're entering, for example, a storage facility, when you walk in there from a physical security perspective, you have to typically log in, right? Who it is, how long you were there. It's the same thing with cloud storage. If those services are disabled, you don't know who entered, who accessed, and it becomes really difficult to understand what has happened in your environment should there be any type of breach.

Dave Bittner: [00:08:23] Now, one thing that you're doing here that's quite useful is you're tracking some of the changes that have occurred since the last time you reported on this. Can you take us through some of the things that you're looking at here?

Matt Chiodi: [00:08:35] Absolutely, yeah. So I think, you know, one of the things that we try to keep consistent between the Cloud Threat Reports is what has changed, right? So we try to, you know, obviously look at some new angles like we did in this most recent Spring 2020 Cloud Threat Report, which was the big focus on DevOps practices and infrastructure-as-code. But certainly, one of the things that we looked at is, well, just how are things either changing, staying the same, or hopefully, you know, not getting worse.

Matt Chiodi: [00:09:00] But what we found in this report was – so, SSH, right? SSH still remains the primary protocol by which administrators get into remote systems, right? So, SSH operates on port 22. What we found from this research here is that about 76 percent of organizations are exposing SSH to the entire internet. That is not a best practice, right? Because in this case here, if I expose in SSH server to the entire internet, that means that anybody can attempt to access it. Anybody can attempt to brute force it. And in terms of trending, we found that this number was actually up by about 20 percent compared to our last report, which I believe came out in July of 2019. So unfortunately, the number around SSH is trending in the wrong direction.

Matt Chiodi: [00:09:54] Now, you know, SSH tends to be very much weighted towards Linux/Unix-type systems. Certainly there's ways to port it over to Windows, but from a Windows server perspective, Windows servers are typically – administration is typically done over RDP or port 3389. And when we looked at that statistic we found, unfortunately, that's up even more. That's up about 30 percent compared to our last one. So just about 69 percent of organizations are exposing RDP to the entire internet. So again, neither of those things are – we would call those worst practices. Those are things you should not be doing.

Matt Chiodi: [00:10:32] Now, on a positive side, one of the things that we looked at in our 2019 report was the use of TLS, or Transport Layer Security. And a lot of times, you know, if people aren't that familiar with it, it's usually if you look up at your browser, if you go to a bank site, you see that little lock – that's what really what we're talking about there, that secure lock. What we had found was that, you know, we looked at the different versions that are in use out there, of TLS. And this is the good side, right? So, we found that this time, only 27 percent of organizations are using outdated versions like TLS 1.1, right? So, 1.1 was abandoned all the way back in 2008. What we found in this report was that number, thankfully, was down about 34 percent. So hopefully, our goal is – and hopefully this is what we'll see when we do our next report sometime the fall – is we'll see that number hopefully be down in the teens, if not even lower. This is a really good thing, both from a customer security and a privacy perspective.

Dave Bittner: [00:11:34] When it comes to those numbers going up for your SSH and your RDP, do you have any speculation as to what would be driving such a noticeable increase in that?

Matt Chiodi: [00:11:46] You know, I think, you know, all of these things we like to really sit back and try to think, OK, what could be the case? And I think that when we look at what has been put out there in terms of services by the cloud service providers, up until even just a few months back, there really wasn't any kind of commercialized offering ways to remotely administer your environments. And so, I think a lot of organizations were really building their own – what we would call either a bastion host or a server where they can kind of get it and then access the rest of their environments. Why it went up from the last report, I would say we don't have a really good view into that.

Matt Chiodi: [00:12:23] But I do believe that what we're going to see to the next report, I think we're going to start to see these numbers go down. Simply because, for example, Azure came out with their Azure Bastion host service, which is now – they're offering a secured, locked-down way that they secure it, right? They offer this as a service to their customers. And I think now we will hopefully see organizations start to move over to those services so they don't have to manage it themselves or not managing the patches of it. So, our goal is, and our hope is, that over the next couple of months, organizations will begin to adopt these types of services, and that should then have these numbers come down.

Dave Bittner: [00:12:59] Hmm. All right, well, Matt, stand by. I'm going to bring in Ryan Olson here. He is also from Palo Alto Networks, from Unit 42. We're gonna switch gears a little bit and talk about your 2020 Unit 42 IoT Threat Report. Let's start in a similar place here. What prompts the creation of this report?

Ryan Olson: [00:13:21] Yeah, so last year we acquired a company called the Zingbox, who is focused on IoT. What Zingbox does is identify network traffic and look at network traffic to try to identify the devices, the IoT devices, that are actually creating that traffic. So they can look at packets that are passing by and say, this looks like a camera, or this looks like a medical imaging device. And then based on that information, they classify them and help their customers put them into the right security model, so they can say, hey, this is a device that doesn't necessarily need to go and talk to the entire internet, and make changes in that way. We've been integrating Zingbox's technology into the Palo Alto Networks's next-gen firewall over the last few months, and what we decided to do is look at the data that they'd collected in 2019 from about 1.2 million IoT devices that were in enterprises and a lot of them in medical situations, and break it down to understand what kind of vulnerabilities do they have and really what kind of security posture all these devices seeing inside these enterprise and hospital environments.

Dave Bittner: [00:14:22] Yeah, you've got some really interesting insights to share here. Take us through some of the key findings.

Ryan Olson: [00:14:28] Yeah, so, a lot of this story around the current setup for IoT is things that are just insecure by default, and in general, people who don't have good visibility into the devices they're actually deploying inside their network. So, a couple of the stats that we found especially concerning, but shouldn't be super surprising to anyone, the biggest one was the number of devices that are now running out-of-date operating systems, especially in the IoT medical-imaging world. What we found was 83 percent of medical imaging devices – these are things like CT scanners, MRIs, x-rays – 83 percent of them are now running an out-of-date operating system, an operating system that's no longer supported.

Ryan Olson: [00:15:11] And that number kicked up massively, by about 56 percent, in January when Windows 7 went out of support. When that changeover happened, it meant that all these devices which are running Windows 7 – a lot of people don't really think about the fact that a medical imaging device or other kinds of IoT devices are really running – they're just little computers, sometimes big computers, and they're running normal operating systems and they can interact with other network systems in the same way. By running this very out-of-date system – so out of date that it can no longer even get updates from Microsoft – it puts them at a significant risk. And there's a lot of reasons why they run these old operating systems, but the issue is just becoming larger and larger as more of them become out of date.

Dave Bittner: [00:15:56] Hmm. Another interesting stat you shared is that nearly all of the IoT device traffic runs unencrypted.

Ryan Olson: [00:16:05] Yep. So, 98 percent of the traffic we identified was unencrypted, meaning it's not running over TLS like Matt was talking about before. And while you might not think about that being a big problem – presumably a lot of this traffic is just staying inside the network, so people aren't sniffing it like they might be banking credentials – all of it is potentially compromisable at that point. If someone was able to sniff it, or someone was able to sort of become a man-in-the-middle, they can make modifications that traffic.

Ryan Olson: [00:16:34] And I think it speaks more to the underlying development process for these IoT devices. Most of the traffic we see across the internet is SSL/TLS encrypted at this point, but because the developers of these IoT devices aren't making that choice in the vast majority of cases, it speaks to the fact that they're not thinking necessarily about the security of these devices from the beginning. That's obviously clear from running these old, out-of-date operating systems, but even more clear by the fact that they're choosing not to even offer this base level of encryption on the traffic.

Dave Bittner: [00:17:09] Yeah, one of the things that you track here in your research are the top IoT threats. Take us through what you're seeing here.

Ryan Olson: [00:17:18] Yeah, so we see a lot of different threats impacting IoT devices. And it really comes down to either insecure operating systems and software – meaning people don't have things patched, they don't have updates being applied to them – or insecure configurations by default. So a lot of IoT devices also have insecure passwords by the way they're set up. Wireless routers, cameras, other kinds of devices you'll typically find in enterprise – by default, they simply don't have a secure configuration, initially.

Ryan Olson: [00:17:48] On the flip side of that, we're seeing attacks as well against these devices. And they're the kinds of attacks that you'd expect to see against normal computers – it's just impacting these IoT devices. So, things like ransomware, backdoors that are being installed on the devices, malware that's being installed so it can be used for launching DDoS attacks, things that are being used to mine cryptocurrency – all of these threats that we generally think of applying just to computers, they also apply to these IoT devices that are deployed all over networks, oftentimes in larger numbers than actual computers inside an enterprise.

Dave Bittner: [00:18:23] Where do we stand in terms of organizations having a good inventory of all these devices – what they have and what's running on them?

Ryan Olson: [00:18:32] You know, that's actually the biggest challenge that most organizations are dealing with when they're starting to think about IoT. When I talk to a customer about their IoT security, oftentimes just having that inventory is the first step. And oftentimes they have an inventory, but it's a inventory based off of what they recorded at the time someone went and deployed it. So, if the folks who are deploying these, you know, thermostats or cameras or IP phones, whatever they might be, if they record, hey, we've got this deployed, we've attached it to the network, then they have an inventory the same way that they're tracking, you know, how many cars the company might own.

Ryan Olson: [00:19:08] But the problem with that is, nearly anyone can find a way to add an IoT device to a wireless network, which means there's all these devices that are being added on a regular basis which aren't being tracked by the IT folks. They don't know what exists. You don't have to go through the process of setting up a complicated network connection and getting a MAC address whitelisted. It's just attach it to the wireless network and suddenly it's part of your network. So the first step for everybody, as they sort of start managing their IoT devices in the lifecycle of these devices, is getting that accurate inventory and understanding what you actually have inside your network.

Dave Bittner: [00:19:45] Well, Matt, I'm going to ask you to jump back in and join the conversation here. You all found some interesting crossover between these two reports.

Matt Chiodi: [00:19:56] We did, yeah. And I think what we all need to remember is that the vast majority – so, IoT is made up of two different things, right? You've got the software that runs on the IoT device, and then you have the hardware. From a software perspective, most of that IoT software is being developed in the cloud, right? And the infrastructure – so, if you have an IoT device, it doesn't matter what it is. Like Ryan said, whether it's a thermostat, a camera – most of that data that's being transmitted is coming back to some type of cloud service. And that's really where that interchange is, right? So, certainly the TLS traffic we were talking about, there's an interplay right there.

Matt Chiodi: [00:20:35] A lot of times some of these IoT devices – they don't have the same level of processing power that, say, a laptop or a server might have, in which case they may be using that lower level of encryption, if they're doing it at all. And then in terms of the software that's being developed, a lot of that infrastructure is running on cloud. And so, you have to look at this, not just at the device, but the entire ecosystem that's surrounding the IoT device, right? So, again, that backend cloud infrastructure.

Matt Chiodi: [00:21:06] And from what we found, is also in the report was that, you know, organizations are just not yet embracing the concept of DevSecOps. So, organizations have talked about it. We've talked about it a lot as an industry. But most organizations are still not to the point where they've truly wrapped security into the development process, because if they did, we wouldn't be seeing these numbers either on the IoT side or on the cloud side, right?

Matt Chiodi: [00:21:34] And so, it's our contention that, you know, when organizations – all they need to do from this perspective, is, you know, when you talk about even just starting building your cloud environment, organizations need to shift how they're doing things, right? And so we've talked about this whole concept of "shift left," which is just kind of the very beginnings of moving towards DevSecOps. And all we mean by "shift left" is literally about moving security to the earliest possible point of the development process. So, this begins by empowering developers, giving them plugins that work directly in their integrated development environments so that if they are starting to create a piece of code – whether maybe it's a new container they're creating, right, some type of new micro service or whether it's a new infrastructure-as-code template that they're creating – that they get early feedback that, hey, you're introducing a new vulnerability.

Matt Chiodi: [00:22:24] So that's kind of step one. And then obviously, as that software moves through the development lifecycle or through their CI/CD pipeline, that there are also other checks built in where IaC templates are being scanned, new containers that are being created are being scanned. New – if they're using serverless, right? In a lot of cases, it's not even containers anymore – it's serverless. Making sure that there is scanning throughout that process.

Matt Chiodi: [00:22:48] And then making sure also that standards are being enforced, right? So, anytime that an organization, whether it's a startup – Right? A lot of times these IoT devices are being created by startups – or whether it's an enterprise organization or, you know, a large government organization, enforcing standards, security standards, becomes extremely important. And anytime that you work at cloud-scale, it requires that strict enforcement of standards. And so, what we like to recommend, if an organization, you know, doesn't have a cloud security standard, that they check out what the Center for Internet Security, or CIS, has created. CIS has a benchmark available for all the major cloud platforms. They have benchmarks available for multiple different operating systems. It's a great way to start.

Dave Bittner: [00:23:34] You know, Ryan, we often talk about defense-in-depth. And when I think about the interplay between IoT vulnerabilities and the cloud, it sort of strikes me as – there's a potential there for a cascading vulnerability-in-depth situation.

Ryan Olson: [00:23:54] Yeah, it certainly exists. So, when you have devices that are communicating with the cloud, if those get compromised, they may be a route into the IoT devices in the same way that compromising an IoT device may be a route further into the network. Once again, people not thinking about the fact that these are little computers that can run all sorts of code, whatever an attacker might want to. If they are inside your network and they're not isolated from other devices, that makes them a risk of coming into the network and being a route in, as well as being at risk from other things happening inside the network, especially looking at the medical area.

Ryan Olson: [00:24:28] One of the things that we were investigating was the types of devices and what kinds of VLANs they're on inside of hospitals. So when you see things like medical imaging devices and infusion pumps – which are connected to the network because they're sending data back and forth and they're sending images and other things – if they're on the exact same VLAN as a doctor who's sitting down at their computer and opening email and potentially clicking on phishing emails and opening malware, that's problematic, because if their Windows or Mac computer, it gets infected with malware, it might be able to spread to those other devices in the network. And that's just completely unnecessary. These devices don't all need to be able to talk to each other all the time.

Ryan Olson: [00:25:10] And one of the stats we actually pulled for this report was the number of devices that are on separate VLANs so that they can't communicate with each other. What we found was about 72 percent of the VLANs in healthcare environments in particular have a mixture of these IT and IoT devices, which means you do have that crossover between one device to another. Based off of the data from this year's report compared to what we had looked at in the past, that is going down. We are seeing more and more VLANs and people are separating these devices and healthcare environments, but it's still not happening nearly at the level that it needs to, because you simply don't need all these devices to be able to talk to each other in the same way that you might need other kinds of normal IT devices to do so.

Dave Bittner: [00:25:57] That's Ryan Olson. He's VP of Threat Intelligence at Palo Alto Networks. He also leads the group they call Unit 42.

Dave Bittner: [00:26:05] Our thanks to Matt Chiodi and Ryan Olson from Palo Alto Networks. The research we discussed today was the Cloud Threat Report from Unit 42, as well as the IoT Threat Report, also from Unit 42. We'll have links to both in the show notes.

Dave Bittner: [00:26:20] Thanks to Juniper Networks for sponsoring our show. You can learn more at juniper.net/security, or connect with them on Twitter or Facebook.

Dave Bittner: [00:26:28] And thanks to Enveil for their sponsorship. You can find out how they're closing the last gap in data security at enveil.com.

Dave Bittner: [00:26:36] The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Elliott Peltzman, Puru Prakash, Stefan Vaziri, Kelsea Bond, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Peter Kilpe, and I'm Dave Bittner. Thanks for listening.