Staying Ahead of Cloud Attacks
David Moulton: And is honeyified an official term?
Nathaniel Quist: It should be, right?
David Moulton: Right, like it got me laughing. I'd muted the mic, but I didn't - I've never heard honeyified. I like that. I like that, that's something that we should have in more of our headlines. Things that have been honeyified.
Nathaniel Quist: I think more things. You know, honeyified, it's like a tasty treat.
David Moulton: Right.
Nathaneil Quist: You just want to grab it, but you know.
David Moulton: Yeah, sounds like breakfast. [ Music ] Welcome to "Threat Vector", the Palo Alto Network's podcast where we discuss pressing cybersecurity threats and resilience and uncover insights into the latest industry trends. I'm your host, David Moulton, Director of Thought Leadership. [ Music ] Today, I'm speaking with Nathaniel Quist, Q, those who know him. He's a manager of Cloud Threat Intelligence at Palo Alto Networks and Cortex/Unit 42. Q leads the team focused on uncovering and analyzing threats targeting cloud infrastructures including AWS, GCP, and Azure. With his expertise in public cloud security, threat intelligence, and malware analysis, he has been at the forefront of understanding the growing and evolving cloud threat landscape. In this episode, we'll explore the intricate and hidden world of cloud threats, including the large-scale cloud extortion operation recently uncovered by Unit 42, and discuss the emerging threats and challenges organizations face when securing cloud environments. Here's our conversation. Q, welcome to "Threat Vector". I'm really excited to have you here.
Nathaniel Quist: Awesome, thank you, David, appreciate being here.
David Moulton: Talk to me a little bit about your background in threat intelligence and maybe what excites you the most about working for Unit 42.
Nathaniel Quist: Threat intelligence, I was one of the fortunate few that actually started my career in threat intel. It's kind of hard to find, usually we start in like IT or sys admin or something of that nature. But I love intel. I love tracking threat actors and how they perform those operations. So, getting the opportunity to work with Unit 42, you know, some of the smartest minds that I've ever met with are here. And so, being able to bounce ideas, like I see people doing this or I see, you know, systems that are doing this, you know. And have that back and forth camaraderie with a really awesome group of people, it's pretty amazing.
David Moulton: Yeah, passion and intelligence are not in short supply in Unit 42, I've noticed that myself. Today, we're going to get in and talk about the cloud threat landscape and some of your thoughts on the latest developments, including cloud extortion operations, cloud native threats, and how businesses can strengthen their defenses. We've got a lot to talk about. So, Q, let's look at that research that you've recently published on Unit 42. I was highlighting some of the large-scale cloud extortion operations and we'll have a link to this article in the show notes. But could you walk us through some of the key findings and how those attackers were exploiting cloud environments?
Nathaniel Quist: Yeah, most certainly. So, it started out with exposed credentials. And these exposed credentials were actually environment variants. Just a quick idea, a quick synopsis of what environment variable file is, let's say you have a web front end system, and you want to be able to connect to a database in the back end, and you need to have authentication to make that happen. You don't want to have to manually do it because it's automated, so you have something called an environment variable. And that will have your username, it'll have passwords, perhaps session tokens, things of that nature. And due to the sensitive nature of these, I obviously don't want those to be exposed publicly, however, these were. And so, the threat actor found that there are a number of environment variables that were exposed. They were specifically looking for something called Mailgun which is an email service, which is a very common target for threat actors. Email services within cloud because they want to send phishing emails, that's really important. So, they were looking for Mailgun. As they were looking through Mailgun, they started collecting a lot of credentials. They targeted about 110,000 different domains and IP addresses looking for environment variables, they were able to collect 90,000 plus environment variable files. And inside of those environment variable files, they were - they found roughly 1200 AWS access keys and IDs that they were able to get into the cloud systems from there. So, it's kind of worm-like in some aspects when they were able to come find one environment variable, they were able to get into that S3 bucket behind it. And then that RDS system, they pulled down and deleted all of those particular files from S3 and RDS, left it a little ransom note, and said you know, we have your system for ransom, please pay, you know, pay this amount. And then, you can have your data back. And so, that's kind of where it started. It was a real, real attack that we went through, we were working with the incident response team for Unit 42. And they pulled us in for, you know, threat intel enrichment to see if we see where they're coming from, how that worked.
David Moulton: So, how does an attack like this evolve over time? And are there specific patterns that you're starting to notice?
Nathaniel Quist: Well, what we're finding with threat actors, specifically looking at cloud, they're really starting with that low hanging fruit, you know. A couple years back, it was all cryptojacking and crypto mining operations. They're starting, we're starting to see an evolution into ransomware because, you know, a lot of the just exposed systems are not really there anymore to make cryptojacking super lucrative. So, they're looking for exposed of any access management credentials, those keys and IDs, session tokens, so they can get into those cloud environments. Once you're inside of that cloud environment, we're starting to see more evolution in lateral movement. We're starting to see more privilege escalation happen in those environments. And then that makes, you know, ransomware attacks, theft of intellectual property, a little bit more common. So, the evolution is really, as we as, you know, threat researchers and intelligence, you know, analysts, we're seeing that these types of attacks are occurring. We create blocks and mitigations for them. Threat actors will have to raise their game which, so they can get over some of those protections to find more credentials or more ways to laterally move or increase their permissions inside of that cloud environment. And then, you know, we defend against that. And then, they go over that and they go over that. So, we're getting to that cat and mouse game.
David Moulton: Right.
Nathaniel Quist: So, really what we're seeing last few years, very exposed endpoints. A lot of exposed Docker containers, a lot of exposed Kubernetes containers, a lot of exposed, you know, virtual machines in cloud environments, they would just take advantage for cryptojacking purposes. Now, we're starting to see they're starting to leave the endpoints, go into the cloud system, and start their own endpoints to do their operations. [ Music ]
David Moulton: Q, one of the challenges that you mentioned in your research is the difficulty in detecting cloud-based extortion. And what makes cloud threats more difficult to detect and mitigate compared to traditional on prem threats?
Nathaniel Quist: Yeah, I think it's really, it's a matter of volume in two ways. In one way, as researchers with Palo Alto, you know, we have visibility into telemetry from our customers' environments, things of that nature. We are really looking for those granular events, you know, what are those one-time operations that are happening. Are there, you know, specific pieces of malware running in those cloud instances? Are there, you know, particular operations of discovery, either identity access management credential discovery or other file discovery, other system discovery. Some of those operations really require a one-time telemetry agent on those systems to be able to detect those things. Within cloud systems, the majority of cloud organizations or who are using cloud operations or cloud environments, they have so much data that they're dealing with. They are dealing with posture, they're dealing with, you know, proper architectural control, things of that nature, that sometimes if you add more run-time telemetry on top of that, it just makes it, it just like blows the roof off the ceiling as far as data is concerned. So, it's difficult to assess and difficult to address. You know, do you have to have a run-time agent on every single cloud instance that you have in your environment? That's a lot of data. It could also be very expensive. So, there's some pushback in terms of visibility that we're kind of limited in the amount of visibility in cloud when it comes to that granular run-time operation. And that makes it difficult to detect some of these types of attacks. You know, and at the same time, we have so much data that is coming in, especially from our cloud service providers. I've talked about AWS, you know, but GPC, Azure, IBM, and Oracle, they all had these cloud systems and when you create a new instance, you're going to get several logs like new instance created, you know, associated identity access management credential associated with that, you know. It has these security groups or network access controls. You know, all of this stuff gets loaded into it and that's another log that has to go through that system. So, being able to parse all of that data in an effective, usable, specifically for a security operations standpoint, it gets very, it's very onerous, it's very burdensome in some of those aspects. So, so, data, just the volume of data makes it difficult to parse through all of that. And then, in order to detect some of these more sophisticated attacks, because cloud is a very complex system inherently, you have to be able to have the understanding and the awareness to know that this type of run-time event is important and that we should create an alert upon it. So, it's a matter of volume and volume fatigue, and then sometimes alert fatigue plays into that as well. So, yeah, it's both too much data and not enough data.
David Moulton: Yeah, it seems like it's a bit of the double whammy. And then, you mention the cost and I assume storing all those logs and having all that data hit throughput ends up being one of those things that becomes a business decision, you know. You can spend all of your capital in one area versus another.
Nathaniel Quist: Yeah, especially and then if you have, like, a multi-cloud environment where you're renting, you know, AWS for front end systems and GCP for back end systems, you know. And you have, you know, an actor that will traverse both of those or in a particular operation. You have card costs in both, you know, cloud logging costs in both different cloud service providers. That could be an extra challenge there as well.
David Moulton: Q, from your experience, what are some of the common misconceptions that organizations have about securing cloud environments, especially as they transition away from those traditional infrastructures.
Nathaniel Quist: This is, could be a challenging question in a lot of cases, both from an architectural perspective and then also from, you know, like actual run-time, you know, defense perspective. Some misconceptions are, is that I can just lift and shift everything from my data center up into the cloud, and we'll be, everything will be great. I'll have cost savings and, you know, cloud source will be cheaper and more secure. So, it'll be easier. It's definitely a misconception using cloud. Cloud is very, very good at being dynamic. It's very good at being scalable, but it's only good if you kind of design your systems to work in the cloud effectively. You know, so, there's the misconception is as far as like cost reduction, if you just simply lift and shift into the cloud, you could have a pretty hefty bill at the end of the day, especially when it comes to logging. But when it comes from a security standpoint, there is something that all of the cloud service providers really, really highlight and that's the shared security model. So, you know, the shared responsibility matrix of the cloud. And essentially, cloud platforms give you everything you need to run your operation. They'll give you the hardware in which your virtual machines will run in. Everything on top of that is your responsibility, your - the need for you to create, you know, least privileged identity access management policies or rules so your users can stay within their guardrails, making sure that your systems are patched and up to date. You know, your individual applications, that's the user of the cloud, that's your responsibility. You know, AWS, GCP, Azure, they will not patch your systems for you or they will not design your user access. That's your responsibility. So, that's kind of a misconception that, yeah, the cloud infrastructure is secure, but how you access that particular application, that's your responsibility and that's something that we've seen a lot of organizations be like, oh I'm in the cloud, supposed to be safe and secure. But how you use it is your responsibility.
David Moulton: So, costs, if you're not careful. Security, if you're not careful and thoughtful, those are some of those misconceptions. Anything else? Anything else that comes to mind?
Nathaniel Quist: What's interesting about cloud is it's very wide and it's very deep, you know. And it just depends on how complex you want to get, you know. A big piece would be, you know, the difference between agent and agent lists security tools is very important to know and differentiate. Agent lists will give you asset control and inventory management and patching, vulnerability patching capabilities. But it won't give you that run-time operation, what is happening on those particular systems or those endpoints. That's where you need to have that agent installed either on your container hosts or on that virtual machine itself. So, it's recording, you know, the run-time capabilities that that cloud system is having. That would be the distinction to an agent and agent lists. They both work parallel and hand-in-hand, and you don't need to have an agent like installed on every single cloud resource, but your highly critical ones, your, you know, your front end web servers, your database systems, your, you know, the crown jewels, so to speak, those should most certainly have an agent installed on them.
David Moulton: So, any article that you all published, you also discussed this idea of cloud native threats. And can you explain what makes a cloud native environment such as those using Docker, Kubernetes, particularly attractive to attackers?
Nathaniel Quist: So, cloud native threats are interesting in that they, they are first and foremost only looking at cloud operations. There is, I think we can all agree, that there is a big difference between on prem organizations and how they're structured and managed and cloud organizations and how they're structured and managed. And there's a lot of correlation and, you know, similarity between the two, but cloud, it speaks the language as its own way of performing operations. So, if you have a threat actor who is well versed in how cloud systems and resources and services speak to each other, it's going to be very, you know, they know how to hide in the noise, so to speak. So, when you have threat actors who are well versed in like container escapes or being able to traverse identity access management and they may have automation, their own scripts that they've built, and their own tooling that they've created to specifically highlight and leverage those capabilities, you can see the operational scope increase dramatically with a cloud threat actor. In the environment variable piece that we put out, that entire operation was, you know, automated in one form or another. There's only two or three points were we able to find that an actor actually physically touched a keyboard to make something happen or to push the operation in one way or another. So, from initial access to exfiltration or usage of data, I mean we're looking at that dwell time being, you know, minutes to hours instead of, like you know, weeks to months.
David Moulton: And are you seeing that more and more cloud attacks are actually run against a playbook like this with full automation or very little human interaction?
Nathaniel Quist: Yeah, most certainly. So, within my team, we created something called the honey cloud and the honey cloud is a, if you're familiar with the honeypot which is basically an exposed application that anything that happens inside of the honeypot should be deemed malicious because there should have no legitimate value whatsoever. We created a honeypot, but on a cloud scale. So, we could have dynamically expanding and, you know, auto generative systems within cloud environments that's all, it's all honeyified, right? We released an identity access management credential inside of the GitHub repo and then, within four minutes of that identity access management credential being placed inside of that GitHub repo, we had active crypto mining happen in our environment. So, it's just a public hub repository, just posted that credential up there, four minutes later, we had active operations happening inside of the cloud environment. That included discovery, privilege escalation, lateral movement, and then finally, execution of a malicious operation. So, automation matters, right? Yeah, automation and scale and to be able to go from, you know, having no access to doing all of the, you know, discovery and automation and attack operation. Within four minutes? That's the power of the cloud, but also from a cloud threat actor, it's pretty significant, pretty serious.
David Moulton: And is honeyified an official term?
Nathaniel Quist: It should be, right?
David Moulton: Right, like it got me laughing. I'd muted the mic, but I didn't - I've never heard honeyified, I like that. I like that, that's something that we should have in more of our headlines, things that have been honeyified.
Nathaniel Quist: Yeah, I think more things, you know, honeyified, it's like a tasty treat, you just want to grab it.
David Moulton: Right, yeah.
Nathaniel Quist: But you know.
David Moulton: It sounds like breakfast. Q, given the level of automation, it seems security teams will find themselves overwhelmed. How can automation and cloud native application protection platforms play a role in lightening the load for the defenders?
Nathaniel Quist: Yeah, definitely. So, automation is, it helps the attackers move in a very specific direction. From a defender's aspect, we can use automation to, again, secure our base, secure our posture, right. So, using infrastructure as code so, those things like terraform or cloud formation in AWS where you are, you know, you have scripted the ability to build infrastructure in cloud, you can scan those templates for misconfigurations, internal databases that have been exposed publicly, you know, within that particular template, you can scan for that and make sure that you're not inadvertently exposing, you know, either hard coded credentials, particular sensitive systems. You, obviously it is scanning for vulnerabilities, something that we all know in the security industry very well is your vuln scanners. But you can also automate the operations that happen, the detection of operations. Within your cloud service provider, you might see an alert that is, you know, a connection from an IP address. And then, it has the creation, the same instance does an API call to your cloud system for a new user to be created. And then that new user has a new role that's been attached to it, and there's a number of steps that go through that. You can look for those behavioral steps, you can use automation to look for trigger points, multiple trigger points over time to alert you if something were to happen in an environment that you want to pay attention to. So, we can use automation in terms of playbooks like I just described there to deliver more data. Like this particular IP address communicated with my end point, what else does it have? Or what else historically has it done? Has it been associated with malware or has it had some sort of phishing operation attached to it? All of that automation can become meta dated influences and help you respond more effectively and more accurately to whatever security risk that you're currently facing. Or potentially need to investigate more of. [ Music ]
David Moulton: It seems to me that cloud attacks are becoming more sophisticated. Are there emerging technologies or strategies that cloud security teams should be aware of to stay ahead of these attacks?
Nathaniel Quist: There are a lot of security start ups in the world right now. And there's a lot of new tools and new technologies that are coming through. Yes, cloud threats are becoming more sophisticated, but I think we have the security teams and the personnel that we have operating and looking at this on a daily basis. They're also becoming more sophisticated and more interconnected. And we're working really, really well together. So, I don't want to say just throw more tools at it, that's the last thing we need to do. We need to have everything more easily, you know, pulled to the surface. You know, how do we do that necessarily? I think, you know, AI, it's a buzzword, but it's actually really important to sift through massive amounts of data very quickly to pull out very important highlights. Why out of all of this data, you know, these are the individual things, this is the common pattern between all of these pieces that we should look at. AI is really good at sifting large amount of data and making it human readable at the end of it. You can make your own LOMs to help, you know, parse and solidify data within Palo Alto, the XIM process is really good. It takes petabytes, literal petabytes of data and distills it down, so you can quickly pull out important markers or security alerts in your environment. So, that is a technology that we should definitely have more insight into. And then, also we should really focus a whole lot more on how containers are running and how to save containers. We are, as an industry, using containers more and more and more frequently. And we need to have a better way, as an industry whole, to understand how to forensically save that data. So, researchers like myself or the researchers and the teams that who's listening to this podcast, they can say, oh this container was compromised because of this. We didn't just blow it away and we don't have that evidence anymore, we can save that snapshot and then, we can look at it and say, this is where the vulnerability happened or this is where the, you know, the incident took place.
David Moulton: Q, if I play this back right, what you're saying is that we'll spin up a container, something will then go awry, we don't necessarily know that right off the bat, but then we kind of do the job of sweeping up the evidence and throwing it away before you can come in and investigate what happened.
Nathaniel Quist: Right.
David Moulton: And what you'd look for is some way of preserving those containers that we've decided we don't need such that you could have a full picture of what went wrong when, and how do we stop that from happening again.
Nathaniel Quist: Yeah, I think you played that back perfectly. The piece that I would really, you know, pull out of that is that, currently, if it, it's current thought that, oh, cloud run-time instance failed. Let's just reboot it. Oh, it's back again, we're back in operation and we can continue doing, you know, our production work. But what really happened is that we're missing is, well why did it fail? What happened, you know. If a threat actor or, you know, a real cloud threat actor who specifically targeted that application and that container and caused it to fail or caused it to do something else, we want to have that evidence. And that evidence is stored in the state of that container and we want to keep it to ensure that we have it. We don't just want to sweep it away, which is like kind of common practice for a lot of dev op organizations right now is, it failed, restart it, okay we're good and we'll just keep cleaning the slate over and over and over again. Which is fine, I mean, it's, you get back production very quickly, but if we can keep that snapshot, if we can keep that history, that evidence, that really does allow researchers to know why.
David Moulton: So, looking ahead, do you see the cloud threat landscape evolving over the next few years, and what should businesses prioritize to future-proof their cloud security strategies?
Nathaniel Quist: Yeah, I mean a big question. My opinion is that we should focus more time on automation and we should really focus that time on automation in the ability to create known state of cloud systems. So, what are the applications and services that we're using and are they created effectively to begin with. So, starting with infrastructure as code templates, making sure that they are, everything is started from a very base, secure state. From an identity and access management policy, you know, really getting organizations to understand that we need to use roles as opposed to individual IDs or individual users in cloud environments. That will make user management a whole lot easier to maintain and operate, you know, to operationalize. And then, for also security teams to know who happened when and where. So, making sure that you're doubling down on automation, simplifying the process of preserving data as it's happening. And then, make sure that you have some sort of seen app operation, some sort of run-time operation to look at your critical assets to ensure that you're seeing everything that they're using.
David Moulton: One of the questions I always like to ask at the end of these conversations is what's the most important thing that a listener should take away from our conversation today?
Nathaniel Quist: To me, the most important thing that organizations should take away, specifically for cloud operations, is the proper security, maintenance, and monitoring of your identity access management credentials. Having them leaked in environment variables that are publicly exposed, having them hard coded to inside of virtual machines or inside of source code that's stored in GitHub or wherever you're storing your source code. Removing the hard coded credentials, it's roughly the equivalent of having a post-it note with your password or username and password to the keyboard. If you know where to look, it's really easy to find identity and access management credentials. So, double down, triple down on identity and access management monitoring and visibility. [ Music ]
David Moulton: Q, thanks for the great conversation today. I really appreciate that you took me through your insights on the cloud threat landscape, including cloud native threats and the large-scale cloud extortion operations your team at Unit 42 uncovered.
Nathaniel Quist: Thank you so much, Dave. It was a pleasure talking with you. And look forward to more conversations in the future.
David Moulton: That's it for today. If you like what you heard, please subscribe wherever you listen to podcasts and leave us a review on Apple Podcasts or Spotify. Those reviews really do help us understand what you want to hear about. If you want to reach out to me directly, email me at threatvector @paloaltonetworks.com. I want to thank our producer, Michael Heller, our content and production teams which include Kenne Miller, Joe Bettencourt, and Virgina Tran. Elliott Peltzman edits the show and mixes the audio. We'll be back next week. Until then, stay secure, stay vigilant. Goodbye for now. [ Music ]
