Misconfigured identity and access management (IAM) is much more widespread.
Dave Bittner: Hello everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.
Matt Chiodi: What we found was that this problem that we had found in this one customer was actually much more widespread. This is something that is likely affecting thousands of cloud accounts worldwide.
Dave Bittner: That's Matt Chiodi. He's Chief Security Officer for Public Cloud at Palo Alto Networks. The research we're discussing today is their second-half 2020 edition of their biannual Unit 42 Cloud Threat Report.
Matt Chiodi: We wanted to really understand the impact of cloud misconfigurations. So, we were actually approached by one of our customers to do a red team exercise on their AWS environment, and as part of that red team exercise – so, one thing I'll say there is, at Unit 42, we don't typically carry out offensive operations, right? We are strictly a research organization. But in this case here, we made an exception because of the sheer scale and size of this customer. So, we carried out a red team exercise, and in doing that, we very quickly discovered two critical AWS misconfigurations in less than a week that could have led to a multi-million-dollar data breach.
Dave Bittner: Hmm. Well, can you give us some insights here? Because I think all of us are aware of the kind of, you know, endless chain of stories that we hear about people just sort of leaving their AWS buckets out there hanging wide open inadvertently through misconfigurations. But I suspect there's some nuance there that we might be missing, you know, with the bigger picture.
Matt Chiodi: Sure, that's a great question, Dave, and you're right. This wasn't a case of a customer having a misconfigured bucket or anything like that. This was something, I would say, slightly more complex. So this had to do – at the core of it – this was not an AWS misconfiguration, right? It had nothing to do with anything that AWS did. This had to do with exactly how the customer had in IAM role misconfigured. So this had to do specifically with identity. And our researchers, in just about a week, were able to find this misconfigured role, and then they were able to exploit it to the end where they were able to get complete and total control over this customer's AWS environment.
Matt Chiodi: So, this one wasn't an S3 bucket up front, but they were able to use this misconfigured identity, this misconfigured role, and then from that, they were able to get to everything that was supposed to be inside this environment. So, yes, they were able to get to S3 buckets, they were able to get to encryption keys, and eventually they were able to own the entire environment.
Dave Bittner: Can you give us some insight as to what happened on the client side? What did they overlook? What caused the error on their end that you all were able to uncover?
Matt Chiodi: That's a good question as well. I mean, I think from our perspective, our assumption going in was, look, this is a massive, massive customer, right? They themselves ran a SaaS platform that they then turned around and offered to their customers. So, our assumption going in was that they were going to have stellar, absolutely stellar security across the board and that we probably wouldn't really find anything. In this case here, where this customer had not done a good job, it didn't have anything to do with patching, right? Their systems, as far as we could tell, were very well patched, things like that. They didn't have any kind of application configurations or application patches that were missing in this case here. It had to do with the fact that they did not have strong identity and access management, governance.
Matt Chiodi: And all that means is that in the cloud – and this, I would say this is an area that is very different from the on-premise world. So, in the on-premise world, when you're talking about identity, you're dealing with usernames, passwords, groups, and permissions. You have those four key elements in the cloud. You have all four of those. But then on top of it, you have this new thing, which is a role, right? And typically, there's just not one role in a cloud account. You could have hundreds in a single cloud account. And that was the case with this customer. So all we happened to find was just one of those roles that was misconfigured. In this case, it had a wild card, it had that asterisk in there. And that simply meant that once we were able to find this misconfigured role, all we had to do was figure out, OK, what else does this give us access to? And that's how we were able to get in and then start to jump around and be able to eventually own the entire environment.
Dave Bittner: And to be clear here, I mean, this is a well-resourced company we're talking about here. These are not folks who are scraping for every dollar to try to do things right. It wasn't a matter of them not having the team and the resources to get it right the first time.
Matt Chiodi: Absolutely. This customer has a large security team. They likely spend millions of dollars a year on their cybersecurity. And again, by all our – everything that we could tell, we were given very limited information right on the environment – as far as we could tell, they were doing most things right. And this just happened to be one area that they simply overlooked.
Matt Chiodi: And this is an interesting kind of intersection as well, that we found, right? So, we do our cloud threat reports just about every six months, and so in February of this year, February of 2020, we did a different exercise where we actually looked at something known as infrastructure-as-code templates. And that sounds kind of technical, but all it means is that, you know, in the old way of building applications in the cloud, you would go in and you had to manually, you know, I want to click here for some storage, click here for some networking. Now, all that's done in a template, or in code.
Matt Chiodi: And so, the way that fits with this current customer that we're talking about, they used infrastructure-as-code templates, and because of that, they actually made the problem worse. Because they had this one misconfigured role, and then they used these templates to create their subsequent environments, they actually replicated that misconfiguration to their other environments. And that is what allowed us to get to all of their environments and own their entire infrastructure.
Dave Bittner: So, what's the lesson here? I mean, for four organizations around the world, what sort of steps should they be taking to make sure that they're not having the same sort of thing happen to them?
Matt Chiodi: I mean, I think from our perspective, kind of what we walked away with is that, for many years, the things that have been at the forefront has been, you know, make sure your systems are all patched. And that is absolutely still a baseline important thing. You have to make sure systems are patched. In this case here, this customer was patched. And we talk about make sure that you've got strong passwords. We do not exploit anything to do with passwords, right? As far as we know, they had strong passwords. What we walked away with here is that organizations, especially those that have great scale in the cloud, they need now to be absolutely focused on identity and access management and the governance of it.
Matt Chiodi: Because in the cloud, this is something that's different, right? So, even in an on-premise world, if a system itself – let's say you have just a server, and let's say that it's missing a patch, and someone exploits that patch. They're able to now get onto that system. The blast radius of what they're able to do is typically limited to just that system. Attacker gets in, they may be able to get on that individual system. And then, of course, they start to look laterally, what else can I see? But that's usually the major the biggest part of it.
Matt Chiodi: Unfortunately, in the cloud, if you have misconfigured identities, or misconfigured roles like we saw in this case here, the blast radius can be much larger. Instead of it just being a single host that can be compromised, in this case here – and I'll talk about the second part of a research in a minute – the blast radius is not just the host anymore. Now it could be the entire cloud infrastructure that's attached to it. So that's kind of one of the big takeaways that we found as part of this research, is that the blast radius and the impact tends to be much greater than it would have been if this was a traditional on-premise environment.
Dave Bittner: Well, let's move on and talk about the second part of your research here. What else did you discover?
Matt Chiodi: So we thought, OK, we carried out this exercise, and as I mentioned, in a very short period of time, we were able to compromise the environment. We wanted to pull back and really try to understand, OK, you know, yes, this was a well-resourced customer, but, you know, maybe they just had a bad day, right? Maybe this was just a misconfiguration and it would never happen again. So we wanted to look more broadly across the industry just to see how big of a problem is it.
Matt Chiodi: And so what we did was, we looked at one of our favorite sources for public data, and that is GitHub. Most people know GitHub. It's a place where developers love to store their code, share code, et cetera, store it there. And what we did is we leveraged GitHub and we downloaded well over a hundred thousand repositories that are publicly available using the GitHub searching API. We then began to mine through that looking for AWS account IDs. So we found, you know, literally thousands of AWS account IDs.
Matt Chiodi: The next step in the process was making sure that they were valid accounts. And so we were able to do that. And then we started to also mine that data to look for those IAM roles that we were talking about previously. And so basically, we have now have a list of valid AWS accounts and a list of the most common role names. And then we just started different combinations trying these out, right? And this is exactly what an attacker would do. They would try to do an attack that maybe is not even directly against a customer, because they might see that, but they get as much information as they can in a very kind of low-and-slow way. And that's exactly what we did here.
Matt Chiodi: And what we found was that this problem that we had found in this one customer was actually much more widespread. In fact, we found literally hundreds of EC2 backups that we would have been able to access had we not been ethical in how we were doing this. We found encryption keys that were available publicly. And there were many accounts that, again, had this not been Unit 42, had this been an actual attacker, that they could have gotten access to. So again, we walked away from this saying, you know, this wasn't an isolated issue with one customer. This is something that is likely affecting thousands of cloud accounts worldwide.
Dave Bittner: How can organizations go about checking themselves out? Is this a matter of an internal audit? I mean, obviously this organization reached out to you to test this. Is it likely that they would have found it on their own through their own processes?
Matt Chiodi: You know, they may have found it eventually. But unfortunately – I mean, fortunately, in the case of this customer, they were able to have forensics involved and they were able to determine that they were not breached. No one else, luckily, had discovered this. But a lot of times, you know, clients don't get lucky, right? So I think in this case here, what I would say organizations really need to focus on moving forward is two things. And we talk about this in the report. Near the end of it, we basically give ten recommendations or best practices for really trying to eliminate these types of things. So I won't go over all ten, but I'd say there's three in my view that really stand out.
Matt Chiodi: The first is just the whole concept of granting least privilege. And all that means is, look, it's about figuring out, you know, here's a developer. What is it that they need to do on a regular basis? And then it's just figuring out how do I give them the ability to do what they need to do and give them the ability to do nothing else. That is the concept of least privilege.
Matt Chiodi: What we see oftentimes, though, is that these accounts for developers, they may start off in some kind of development environment and they start off with a lot of privileges. And then as things get down the road, those privileges never get pulled back. And then that perpetuates itself into then production environments. So number one is to really look for a way to grant least privilege to start.
Matt Chiodi: And then the second one has to do with just – you know, we talk a lot about auto-remediation in the industry. How can you make things automatic? But when it comes to these types of identity misconfigurations, it becomes really important to automatically look for – so in this case here, what this customer is now doing, but should have been doing in the past, is they should have been automatically scanning their IAM roles looking for these types of misconfigurations. And then when you find it, instead of just kind of throwing out an alert and saying, hey, we have a misconfiguration, actually automatically fixing it. These are the types of proactive things that we believe customers absolutely need to be doing, with the speed of DevOps. It must be automated. It cannot be manual.
Matt Chiodi: You know, as organizations scale their cloud infrastructure, a lot of times the focus on the cost of doing that, it might be glossed over. Simply because, you know, when organizations are spending a lot on cloud, a lot of times people don't really catch if maybe there's a couple of thousand dollars more in one month. And the people that the attackers that are carrying out these cryptojacking attacks, they've gotten smart. They don't take up, you know, tons and tons of resources anymore because they know they're going to get caught. And so they specifically design their scripts to actually run at a low utilization. And so, you know, they can oftentimes go for longer periods of times and not get caught in these environments. And again, poorly configured environments are the ones that are most at risk to cryptojacking.
Dave Bittner: Our thanks to Matt Chiodi from Palo Alto Networks for joining us. The research is the second-half 2020 edition of their biannual Unit 42 Cloud Threat Report. We'll have a link in the show notes.
Dave Bittner: The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Elliott Peltzman, Puru Prakash, Stefan Vaziri, Kelsea Bond, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening.
