Dangers of Cloud Misconfigurations

David Moulton: Margaret, last time we tried this, I had issues with neighbors building incredible bits and pieces of landscaping. I wanted to open today though, with a quick question. Do you now have a cybersecurity joke that you want to tell our listening audience?

Margaret Kelley: Absolutely. How did the hacker get away from the authorities?

David Moulton: I don't know.

Margaret Kelley: He ransomware. [ Music ]

David Moulton: Welcome to "Threat Vector," the Palo Alto Networks' podcast, where we discuss pressing cybersecurity threats and resilience and uncover insights into the latest industry trends. I'm your host, David Moulton, Director of Thought Leadership for Unit 42. In today's episode, we're discussing the complexities of cloud security with a true expert in the field, Margaret Kelley. Margaret is a seasoned Digital Forensics and Incident Response Senior Consultant at Unit 42, Palo Alto Networks' elite cybersecurity team. With years of experience as both a Cloud Engineer and an Incident Responder, Margaret has a deep understanding of the challenges organizations face in securing their cloud environments. She's also been a speaker at Black Hat and has a passion for leading teams in cybersecurity. Today, she'll be sharing her insights on everything from common vulnerabilities to advanced cloud threats. Here's our conversation. Margaret, welcome to "Threat Vector." Excited to have you here.

Margaret Kelley: Thanks, David. I'm really excited to talk with you each time.

David Moulton: And today we're going to talk about cloud security and why everything in your cloud should be set to public, right?

Margaret Kelley: Absolutely. May as well let everything be publicly accessible. Makes your life so much easier.

David Moulton: Actually, I think we're going to be talking about securing the cloud and some of your thoughts on navigating cloud threats. And just in case this is all you listen to today, please don't set everything in your cloud to public. Margaret, given your extensive experience in cybersecurity, how have you seen the landscape of cloud security breaches evolve over the years?

Margaret Kelley: So, when organizations were first moving to the cloud and migrating their workloads, what we saw was a lot of basic misconfigurations that led to really large data breaches. So, I still remember reading article after article about, you know, the latest organization that had exposed all of their data to the Internet because they had made their object level storage publicly accessible. Well, luckily, now that is not the headline that we are seeing constantly anymore. But now we are seeing these cloud security breaches where the threat actors are really advanced with their cloud knowledge and that they are using cloud native attack techniques to exfiltrate data as opposed to just taking basic data that's publicly accessible.

David Moulton: And when you say, "cloud native," can you talk about that a little bit more?

Margaret Kelley: Yes, absolutely. So, when I'm talking about cloud native techniques, what I'm talking about is utilizing the cloud in a way that it's not supposed to be utilized. So, a good example of this is if you have a virtual machine feature in the different cloud service providers is the ability to take a snapshot because you need to be able to back up your data, and that's kind of how you do it. And so, what threat actors are doing is using that cloud native technique of actually taking a snapshot, but instead of using it as a backup, which the threat actors are not interested in doing, what they're doing is then copying that backup, that snapshot, to their cloud environment. And that is, you know, what I call a cloud native technique for data exploration.

David Moulton: Are there any patterns or recurring vulnerabilities that have persisted despite the advancements in cloud technologies?

Margaret Kelley: Yes. So, what we're seeing is each of the cloud service providers are continuously improving their default security measures, which is something that is always great to see. But what we are seeing time and time again is kind of the old story of, you know, these organizations not patching their virtual machines and leaving them publicly accessible. And it's a lot of work to make a virtual machine publicly accessible in a cloud environment. You got to click a lot of buttons to say, you know, "Yes, I want this thing to be public," but what you're still seeing time and time again, really unpatched old hosts with all these vulnerabilities on them that are public to the Internet, and this is something that we continue to see within our investigations.

David Moulton: Margaret, do you have a hypothesis on why that's true? Is it playbooks that deploy automatically and they haven't been updated? Is it they've forgotten about those machines and they just persist, you know, scale with those mistakes in them? Is it something else?

Margaret Kelley: So, a lot of times, these virtual machines are made publicly accessible because the people creating them don't have enough cloud knowledge as well as basic networking experience and knowledge. So, someone said, you know, to a random engineer, "Hey, we need someone to spin up our cloud environment. Will you do it?" And one or two people end up spinning the corporate cloud environment, but then those people don't have the proper security background. They're not network engineers, and the easiest way for them to set up the environment is just make these hosts publicly accessible. So, then when they're working in the traditional on-prem environment, they can access these cloud hosts and they don't have to worry about actually engineering complex networks. You can just kind of set it up and they think it's, you know, good enough. But in reality, you have these public vulnerable hosts then on the Internet.

David Moulton: Yes, it seems like ease of use and a little bit of being naive, end up being the cause or the underlying root cause for -- for this pattern that you keep seeing.

Margaret Kelley: Yes, especially when if you think about traditional on-prem environment, there are multiple teams managing that. Like you're not going to have a single person purchase your server, and then set it up, and then spin up the web app, and then figure out the networking for that. You would never have a single person do that. You have multiple teams working together collaboratively to make sure that it is built in a way for, you know, the application, as well as built securely. So, that is not something that is translating into cloud environments and that's something that would be really great to see at some point in the future.

David Moulton: So, with your deep understanding of security hygiene, what are some of the critical yet overlooked aspects of cloud security that still surprise you today?

Margaret Kelley: It's really going back to that basic network security. So, the first one is having a firewall. You have firewalls everywhere on-prem, but that is not something that is being utilized in the cloud. So, in the different cloud service providers, they have different network security groups that you can utilize. That is kind of a base level for a firewall, but those are barely being used, let alone a third-party firewall. We're also seeing, you know, no network segmentation. So, your databases are in your same segment with their virtual machines. And there's, you know, no restrictions on what can talk to what on the network. And we're also seeing, you know, this is really bad security hygiene, but databases are also being made publicly accessible. So, I could kind of go on and on about kind of those basic security hygiene things that, you know, we should -- we should really not expect to see in cloud environments, yet we're seeing them all the time.

David Moulton: Sure. And -- and I'm wondering, what is it about these fundamental gaps? You know, why do they continue to exist even it sounds like in some mature organizations?

Margaret Kelley: So, a lot of times it goes back to the time people have to spin up a cloud environment. So, when you're spinning up these different environments, it takes a lot of time to learn about the different services available in each cloud service provider. That takes, you know, a lot of time out of your day to learn about them and then it also takes a lot of time out of your day to experiment with them, so you actually understand how to use the services. But then beyond that, you're tasked with spinning up a brand-new environment from scratch. And so, without this, you know, level of training and time, people don't have what they need to spin up these environments securely.

David Moulton: Do you think that there's a place for AI to help out an engineer that has a lower level of skill, maybe is lacking in time, lacking in focus, or that AI could be an assistant to make better choices?

Margaret Kelley: Well, something AI could do is help people understand how to build a baseline of secure architecture. So, that a lot of times is something that people struggle with is how to build an architecture that incorporates all the components of, you know, whatever application that you're trying to build. And that is something that AI could help out with is giving you kind of that baseline level of, you know, network diagram for how everything should communicate with each other. And then, you can go in and actually implement that. So, that is something that AI could definitely help out with.

David Moulton: So Margaret, I'm interested in getting a little bit behind the scenes on some of the work. You've probably encountered a number of environments where, you know, these basic security practices are -- are lacking or just don't exist at the level that they should, and have led to some pretty significant breaches. Can you share a particularly memorable case where, you know, something stood out or stands out in your mind with our listeners?

Margaret Kelley: Yes, I have one in particular that I think is super interesting. So, in this one, the threat actor gained initial access because a developer left a virtual machine publicly accessible. Now, what I will say they did do correctly is they locked down those network security groups and so the only thing that was accessible was the port required for this application to work. So, the developer did do that correctly. The network security groups were not wide open, which is great, but unfortunately that application had a known vulnerability that, you know, led to the threat actor gain direct access to the host itself. And so, what the threat actor did was once they gained access to the host, they searched around on the box and were able to find some cloud credentials that the developer then left on the host, because they didn't assume -- they did not think that a threat actor was about to get on their host. And so, what the threat actor did was then use those credentials to spin up brand new infrastructure within that cloud environment using infrastructure as code. So, they had pretty much templatized the entire attack. And so, we have seen this pattern, this attack pattern, numerous times. And so, once a threat actor spun up this infrastructure, they use that to then attack other organizations. And this is great for the threat actor, because they are completely anonymized because they are in a different organization's cloud environment. And they also don't have to pay for any of their infrastructure that they just spun up. So, this is one that, you know, with slight nuances, we see this quite frequently.

David Moulton: So, it sounds like there were a handful of issues: the credentials, the vulnerability, storing things improperly, you know, the expectation that this could never become a problem. And I'm wondering, what are the lessons that a listener should draw from this story to fortify those cloud defenses?

Margaret Kelley: There are lessons that can be learned from all aspects of this attack. One of them is not allowing your developers to create publicly accessible virtual machines. But the big one here, there's kind of two, is you want to make sure to limit your permissions with your cloud credentials. And so, making sure that those credentials are really locked down, and so that they, you know, your developers can't spin up any type of resource they want within the environment. So, that's a big one. And then also, you can lock down your environment to make sure that hosts cannot be made publicly accessible unless they go through, you know, some sort of exception process. [ Music ]

David Moulton: So, I want to ask you about the control plane. But before I get into that, you and I were joking off mic when you asked me, you know, if I knew what a control plane was. And I -- I assume it's the plane that's not the experiment plane. I -- I -- I just throw it back to our audience. But can you talk about what the control plane is before we get into this next set of questions?

Margaret Kelley: Yes. So, it's really helpful to understand the difference between the data plane and the control or management plane when you're talking about cloud environments, and specifically cloud attacks. So, the data plane, you can think about it as the core components of a service, so such as the running of a virtual machine or placing objects into object level storage. The control plane or management plane on the other hand, is the layer above that and it controls the resource creation and different APIs used to interact with the services. So, when you're actually spinning up a new virtual machine or you're making modifications to its attributes, that is all at the control plane. So, if you're spinning up a new database, for example, that is again at the control plane. And a lot of times, we see attacks that people don't really know are even happening because they're taking place at the control plane, and not a lot of organizations have the right security mechanisms in place to even detect activity happening at the control plane at all.

David Moulton: So, control plane attacks are prevalent, it sounds like, and not well defended. Can you go a little deeper on why it's so critical for organizations to pay attention to the control plane? I mean the -- the data plane as well, but the control plane in particular.

Margaret Kelley: So, yes, absolutely. This is a lot more prevalent than people are aware of. So, control plane compromise. And a lot of companies aren't aware of the control plane compromise due to a lack of monitoring and alerting that's configured. And so, they usually can't tell if -- if a threat actor spins up a new virtual machine, because they don't have any tooling in place to detect a new resource. So, there are a lot of tools and technologies out there that, you know, can now detect this activity happening in the control plane, but because organizations are so immature in their, you know, transition to the cloud, they definitely are not utilizing those security controls to the best of their abilities.

David Moulton: Margaret, this is what you were talking about earlier, where they spin up some infrastructure and use that for an anonymous attack and it allows the cost of the attack to be put on the victim and it allows them to operate essentially invisibly. What are some of the strategies that you recommend, you know, to stop these types of breaches from occurring?

Margaret Kelley: So, it's pretty straightforward. The biggest one is rotating long-term credentials. If these credentials are rotated, that will prevent the majority of these attacks. So, usually on a lot of our cases, when we come in and we figure out what credentials are compromised, we look to see how long they've been around. And we're not just talking about weeks or months. A lot of these credentials are years old, and they have just been sitting exposed somewhere on the Internet, and the threat actor just decided to use them. So, just by rotating these long-term credentials, that will have a massive impact on these breaches. We're also seeing a lot of organizations inadvertently leaking these cloud credentials. There are a bunch of different ways that they do this. Sometimes it is through code repositories. Sometimes it is through kind of my story earlier, making a virtual machine publicly accessible, but you have a bunch of credentials on them. We're also seeing environment variable files. They're also public a lot of times. And also, with the evolution into cloud environments, these environment variable files now contain cloud credentials. And so, that is another area that we are seeing these cloud credentials also being leaked.

David Moulton: Margaret, let's shift to the shared responsibility model in cloud security. I -- I know that that can be misunderstood. When it first came out, I know I was at best confused. I was probably wrong about it as I read through and tried to understand it. How well do you believe organizations grasp their responsibility in this area?

Margaret Kelley: I think organizations understand the concept of the shared responsibility model, but don't necessarily understand the implications of it. It is fairly easy to understand that each cloud service provider will protect the physical data center and that they will make sure that, you know, if a physical server is reaching end of life, they'll update it. All of that seems to really make sense to organizations. But what is not really translating for organizations is that they are responsible for securing the actual infrastructure themselves. So, I don't see the cloud as really being that much different than protecting your on-prem environment with your security policies and tooling that you need to have in place. But when you're talking about the cloud, a lot of times those security policies and tooling, they're not being translated into cloud environments. And so, even though organizations are responsible for protecting their infrastructure and making sure it's secure, we are not seeing organizations fully understand that that is their responsibility, that just because you're spinning up a server in someone else's data center, that it's that other person's responsibility to protect your server. It is not. It is your responsibility to make sure that your server is well protected and patched and all of that.

David Moulton: So, what are some of the most dangerous misconceptions that you've encountered? And how do you think those might be corrected?

Margaret Kelley: So, this one is very, very dangerous, but it's -- I also kind of find it a little funny. So, sometimes, not sometimes, a lot of times, we hear that because a resource is in the cloud, it has to be public. So, because you spin up a virtual machine in the cloud, it has to be public. Like because you spin up a database in the cloud, it has to be public. That is absolutely, I think, the most dangerous misconception that we see all the time. And you know, big shocker, it's not changing. We still see organizations that have public cloud resources because they think that it's -- since it's located in the cloud, it has to be public. And that's not the case at all. Your resources absolutely do not need to be public. And you know, just with similar -- similarly to a lot of other security issues, by providing the right training, whether it comes to security training, network training, that are all really big helpful steps in how we can correct this misconception. And you know, enabling cloud native security tooling is also something that's really helpful in tackling these misconceptions, because it'll tell you that, you know, "You have a public resource and it doesn't need to be public, and this is how you can go about fixing it." And so, that is why I'm a big proponent for, you know, any cloud native security tooling because it actually understands each cloud's service provider. And so, it will really help with that baseline level of cloud and security knowledge within cloud environments.

David Moulton: You've seen how permission misconfigurations can lead to some pretty catastrophic breaches. What do you believe are the root causes of these issues and how should we mitigate those?

Margaret Kelley: We are seeing a ton of these massive breaches happen in cloud environments, mostly because cloud credentials are being exposed in some matter and those credentials have permissions associated with them that are completely wide open. So, this permission issue has a couple of different facets. So, the first one is, you know, you have people within your organization, engineers and developers, that need access to the cloud environment to do their work. But a lot of times they really want elevated permissions, so they can avoid any roadblocks, so they don't run into any issues to perform their jobs. So, that is, you know, one way we are seeing these cloud credentials have way too many permissions, because developers and the engineers don't want to run into any roadblocks. We are also seeing that trying to understand cloud permissions isn't as easy as people would think, and it does take a lot of practice and knowledge to be able to fully, you know, narrow down the scope of permissions that these cloud credentials have. And it really takes a lot of time to figure out how all of this works together. So, if you only have one or two people managing your cloud environment, making sure that the permissions are super granular, is not tied to the priority list when they have the engineers and the developers saying, "Hey, I have this deadline on Tuesday, and I don't have access to something in the cloud." So, all of those kind of boil up into having, you know, long-term credentials that are completely wide open that lead to these really large breaches in the cloud.

David Moulton: Margaret, it seems like it's a series of small mistakes that in aggregate lead to massive levels of exposure and therefore your breaches are huge. But it seems like a lot of these things are -- are pretty simple, pretty easy things to do, best practices. Are there any specific frameworks or methodologies that you think would help avoid these pitfalls?

Margaret Kelley: Yes, so my favorite one is the CIS Cloud Benchmarks, and this is a cloud framework that does help avoid a lot of these pitfalls. So, CIS is the Center for Internet Security. But I like this one because it's a lot more technical than some of the other frameworks, and it's a lot easier to turn this framework into something that is actually -- that you can implement. So, it's not so high level that you're like, "Oh, I don't know how this actually pertains to the virtual machine." It does get into some of the details about how to actually build your environment more securely.

David Moulton: Margaret, you've -- you've talked about how the IT teams are overworked. Some of them don't have the proper training for a cloud environment. You know, sometimes it sounds like it's a very small team that's asked to -- to handle the cloud environment, whereas an on-prem team would be much larger, much more specialized. What sort of structural or -- or educational changes do you think are necessary to address the challenges that we face?

Margaret Kelley: So, a lot of it comes down to how you are staffing and building these teams. So, there are kind of two different approaches that can work in this area. One approach is just having a cloud specific team. And so, this team would have to be built up of very skilled people from different teams on-prem. And so, what you would then do is kind of select some of the top performers from the different teams. Give them time to really deep dive into whatever cloud service providers you are using, and so that they can then combine those skills that they previously had from their on-prem jobs with, you know, their new cloud knowledge and that with that, they can build out a much more secure cloud environment. So, that is one approach. The other approach is educating existing on-prem teams about the cloud. And I really like this one because you have to collaboratively work across different teams to then build out that secure cloud environment. And it really mimics how we do security on-prem. But I will say it definitely -- it's hard to execute on this because there are so many people that are then involved. And especially, when you're talking about education, you want to make sure that regardless of the approach that you take, that your entire, you know, technology organization actually understands some of the basics of cloud. So, whether you're moving your entire infrastructure to the cloud or not, having everyone understand some basic cloud terminology, and making sure that there are some common language that people can use to communicate is something that will go a long way when you're talking about security, when you're talking about how to build out the cloud environment. All of these things, when you have everyone be able to communicate across on-prem and the cloud and how things connect, it is definitely something that makes a big difference.

David Moulton: So, as these cloud environments have grown both increasingly complex and more important to organizations, do you think that we're doing a good job with keeping pace with the necessary investments and proactive security measures?

Margaret Kelley: I will say some organizations are really doing a phenomenal job here and keeping pace with the proactive cloud security, but most are not because it takes a tremendous amount of work to keep up with all of that learning. So, there are definitely a lot of practice security measures that organizations can put in place that -- it will help them get ahead of threat actors and lock down their environment. And so, you can proactively look for these security gaps as opposed to just waiting to get attacked.

David Moulton: Are there any non-negotiables and -- that are out there that you'd advise any organization to -- to implement?

Margaret Kelley: Yes, absolutely. So, the first one is making sure that you have a third-party firewall within your cloud environment. So, making sure that you have a firewall protecting the north-south traffic, so in and out of your cloud as well as east-west. So, monitoring activity inside your network and that also helps with segmentation. The third-party firewall is really important because it drills into the applicational level of the network and that is that next generation firewall. And this really helps with network security improvements, because it adds another level of protection on your network traffic. The other non-negotiable is for sure, having some sort of Cloud Native Application Protection platform. So, the CNAP solution. And this really helps with catching those cloud native misconfigurations. And so, that will proactively catch all of these security issues that we just talked about today, and depending on the solution, it can auto remediate. So depending on how many people you have available to help build out and protect your cloud environment, having a solution that will auto remediate is definitely something that will potentially help with -- help with protecting your cloud environment. And then this last one, this is a personal favorite of mine, but something you can proactively do is do actual cloud pen tests as well as purple team exercises. And this helps prepare your security team to make sure that they can actually handle cloud attacks. So, the logs that you're reviewing in cloud environments, they're not the same as the logs on-prem. And so, by actually doing these cloud pen tests and purple team exercises, you are giving your security team first-hand experience digging into these logs that they might normally not dig into on a normal Tuesday. [ Music ]

David Moulton: Looking ahead to the future, what emerging trends or threats in cloud security do you believe will demand the most attention from organizations?

Margaret Kelley: So, when it comes to emerging cloud threats, we are continuing to see threat actors broaden their cloud attacks. This is really including automation and scripting. So, we are seeing threat actors deploy resources in the cloud via a script. And so, the time that it takes a threat actor to gain initial access, spin up resources and exfiltrate data keeps getting shorter and shorter. David, earlier you asked me about how the evolution of AI can -- it has impacts on these cloud attacks. And what we are seeing is that attackers now don't have to write their own scripts by hand. It makes it a lot easier for them to say that they want a script to do XYZ in the cloud, and that can be written very quickly for them. And so, this also shortens the length of the attack because they don't have to do any of the scripting by hand anymore.

David Moulton: So, it sounds like once again, speed is the ultimate feature, either for an attacker or a defender. Who can go faster, win's the day?

Margaret Kelley: Yes, exactly. And these attacks, sometimes they take a couple months to take place, but a lot of times these attacks are done within a span of two or three days, and terabytes of data have gone out the door just in the span of a couple hours of that attack timeline.

David Moulton: Margaret, I always like to ask guests, "What's the most important thing a listener should remember from today's conversation?"

Margaret Kelley: I would love if the listener would particularly remember that virtual machines don't have to be public in the cloud and that permissions are kind of the name of the game in cloud environments. Permissions are absolutely worth the time, understanding and figuring out and trying to lock down. That is definitely something that will have the biggest impact on, you know, stopping these large cloud attacks.

David Moulton: Margaret, thanks for the great conversation today. I appreciate you sharing your insights on evolving threats in cloud security to the importance of addressing permission configurations before they lead to catastrophic breaches.

Margaret Kelley: Thanks, David. I really enjoyed our chat today.

David Moulton: That's it for today. If you like what you've heard, please subscribe wherever you listen and leave us a review on Apple Podcast or Spotify. Your reviews and feedback really do help us understand what you want to hear about. If you want to contact me directly about the show, e-mail me at threatvector@ paloaltonetworks.com. I want to thank our executive producer, Mike Heller, our content and production teams, which include Kenne Miller, Joe Bettencourt, and Virginia Tran. Elliott Peltzman edits the show and mixes the audio. We'll be back next week. Until then, stay secure, stay vigilant. Goodbye for now.