It's still possible to find ways to break out.

Dave Bittner: Hello everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Yuval Avrahami: We are talking about container sandboxes, which is actually a technology for sandboxing and isolating containers.

Dave Bittner: That's Yuval Avrahami. He's a principle security researcher at Palo Alto Networks with Unit 42. The research we're discussing today is titled, "Escaping Virtual Containers." It's part of a presentation that he gave at Black Hat 2020.

Yuval Avrahami: So, even though containers have a tremendous upside in both ability and in performance, ironically, they're not that good at containing. The attack surface they expose really isn't suitable for hostile workloads which are actively trying to escape and actively trying to break out and compromise the underlying host. So a couple of companies came up with an idea and a couple of tools to sandbox containers and apply additional layer of isolation. And the results represented that a blanket is really how can you – even though there is an additional layer of isolation, how can you break out and still compromise the underlying host.

Dave Bittner: So what is the additional layer of containment that you're working with here?

Yuval Avrahami: So, the sandboxing solution I looked at was Kata Containers. Other solutions that you might have heard of is gVisor by Google, for example. Kata Containers was created by Intel. And how it sandboxes the container is it runs each container or Kubernetes pod inside the lightweight virtual machine, and then you have two layers of isolation. So if a malicious attacker tries to break out of the container – of the sandbox, it will have to first break out of the container and then escape the encapsulating virtual machine. So it's really a difficult challenge.

Yuval Avrahami: And the idea here is, of course, you know, if you run a lightweight machine for every container, there is going to be a performance penalty. But the idea is you don't have to do that for every container in your cluster. A really good use case for deploying that kind of solution is looking at the weakest link in your cluster of containers. For example, a Kubernetes cluster, and saying, OK, that container is exposed to the network, is doing a lot of parsing of input coming from the network. It might be vulnerable, it might be compromised. And, you know, once an attacker compromises a container, he might try to cryptomine, he might try to steal secrets. But one of the attacks vectors which is very concerning is that a compromised container, a rogue container, might try to escape and affect the rest of your cluster. And that's obviously very dangerous. So, when you are looking at your cluster and you say, OK, this is the weakest link container, I can choose to only sandbox that container using a container sandbox like Kata Containers, and then apply an additional layer of isolation.

Yuval Avrahami: So, the reasoning behind containerized sandboxes is really – it's really good and it's really an appropriate solution. But what really motivated me to get into this research is I started hearing folks mention container sandboxes as a solution for what I felt like too many problems. They will say, you know, if you have this misconfiguration, just use a container sandbox, instead of, like, fixing the root cause of the issue. So, for me, what I wanted, what I felt was missing in the container security field is really research that shows, you know, even though container sandboxes are great for isolating your containers, you should be aware that it's still possible. It's much harder, but it's still possible to find ways to break out. And you should really be prepared for this kind of event, right? The main takeaway I wanted people to take from this research is that a container sandbox really isn't a substitute for a secure, cloud-native architecture, right? It's one layer, but you really shouldn't get – it's not a license to neglect the rest of your security posture, if that makes sense.

Dave Bittner: Yeah, it does. And so, you set out to escape from this sandboxed container here. Let's go through it together. I mean, what was your approach?

Yuval Avrahami: The approach was take it there one step at a time. We have two layers of isolation to break out. First, the container, and then the virtual machine. So, I started first with the container, and I actually had more confidence that I could find an issue in the container, because Kata Containers – they created, like, a small container runtime for running the container inside the virtual machine. And I was fairly confident that it may have some problems. Not because a knock of the Kata Containers' maintainer – it's just a really difficult job to create a, you know, a flawless container runtime from scratch. And you can look at the, you know, the five years of past vulnerabilities in Docker as an example. It's really a difficult job. And when I started to look into that, I started to investigate which isolation properties were they applying to the containers and I found one that was missing. And I used that to escape to the virtual machine. So that really was the first part.

Dave Bittner: Hmm. And so where did you go from there?

Yuval Avrahami: So, the next stage was to break out of the virtual machine. And here there was a lot of attack surface to look at – the virtual machine monitor itself, the communication between the virtual machine and the host – a lot of stuff to look at. But where I eventually found issues there were in the shared directory between the virtual machine and the host. So, Kata Containers, the sandbox, they use a shared directory to transfer some files to the virtual machine. But because it is shared by nature, the virtual machine can control the files there just as much as the host, right?

Yuval Avrahami: So, I find a way for the virtual machine to plant a malicious symbolic link in the shared directory. And then when the host tried to place some files in the shared directory or do some operations with files in the shared directory, it will actually inadvertently follow those symlinks and do operation on the host instead of the VM.

Yuval Avrahami: And that way I was able to find a couple of attacks that you can launch on the host. And one of them was a denial-of-service attack where you could unmount the entire filesystem of the host, which of course, that really is bad. If you're running multiple containers on one host and then one container is able to take down the filesystem of the entire host, that host really becomes unusable at that point.

Yuval Avrahami: And the second attack, I was actually able to redirect the container image – the container's filesystem – to every path on the host that I wanted to. So, for example, you can imagine that if I redirect the container filesystem over something like /bin, or /lib, it's really easy to gain code execution on the host, because now every binary that the host tries to execute from /bin, I have control over that. So that was really the entire attack put down together.

Dave Bittner: And how sophisticated were your efforts? I mean, how difficult would it be for an attacker to take the same sort of path that you did?

Yuval Avrahami: It will have to be a very resourceful attacker, because the effort that you have to put in to escape the sandbox – it took me a couple of months. So you really have to have an agenda in place, a target that is using the sandbox that you really want to compromise. It's not that easy. It's a chain of multiple vulnerabilities. So that's really never easy.

Yuval Avrahami: But it could have had a lot of impact if it was found by a malicious actor, because some cloud service providers are actually using Kata Containers to separate between different customers. So you can imagine that if you were running the containers – if you're someone like Azure, for example, and you need one container for multiple customers on the same host, you might want something like a container sandbox to separate, you know, to really segregate between those containers. So that one malicious customer can't impact the rest of the containers of the other customers. So, some cloud service providers have chosen Kata Containers as their container sandbox of choice. And you could actually have a scenario where, if a malicious customer was aware of those vulnerabilities or if he hunted for those, he could actually break out of his container and try to affect the rest of the platform, which could have, like, you know, hundreds and thousands of containers from different customers. So that's really where I think that the vulnerabilities I found could really have a large impact.

Dave Bittner: So, what are your recommendations? Based on what you learned here in your research, what do you say to folks who are using these containers and these sandboxes?

Yuval Avrahami: So, my first recommendation is even if you chose to deploy a container sandbox, and because of, you know, all of the security guaranteed with it, a lot of people are just deploying using the default settings, which are often not the most secure ones. So first of all, if you went the way to deploy container sandbox, you know, put in a little bit of extra effort to really configure it using the most secure configuration. And in the example of Kata Containers, those configurations would have stopped my attack. So there is real evidence that those configurations really work. If for Kata Containers, for example, using Firecracker as the underlying virtual machine really improves the security and it hinders, for example, it mitigates the vulnerabilities that I found. Firecracker is an open-source project from Amazon which is really taking off.

Yuval Avrahami: Another thing you can do is really be prepared for the scenario of a workload container breaking out to the underlying host and trying to affect the rest of the cluster. And in Kubernetes, for example, there are a lot of ways you can restrict the privileges of a single node, so that even if a container breaks out and compromises a Kubernetes node, that node wouldn't have too much – it couldn't do too much damage to the rest of the class. The best recommendation I can give is using the node restriction admission control, and you can look that up in the Kubernetes documentation. It really lowers the privileges of the nodes in your cluster, so that even if one is compromised, it couldn't really do too much damage to the rest of the cluster.

Yuval Avrahami: And of course, if you really have the time, put in some network policies, network firewalling, between Kubernetes nodes so that, you know, only a meaningful and approved and, you know, logical traffic can go between nodes, really also helps to deny a couple of attacks.

Dave Bittner: So, in general, I mean, is it safe to say that putting your containers within the sandboxes is a good idea? Is it generally – is the upside, the security upside better than the – I guess, the potential performance hits that you could sustain?

Yuval Avrahami: That's a great question. And I think it really depends on what you're running in your containers. If you're running your own code that you wrote and you're one-hundred percent certain of what's running in the container, I think there are other best practices that you can follow that can improve the security of your containers without, you know, taking the performance penalty. But if you are running code which you didn't write, or if you're a cloud service provider that you're running code of customers which you have no idea what they're running, then I think a container sandbox really is required because, you know, the technology of containers simply isn't enough to secure hostile ones which are actively trying to escape.

Dave Bittner: Is this a situation where people who are working with cloud service providers, should they be asking them how they're doing this sort of sandboxing? Is this a question that they should be asking to see what the level of security is?

Yuval Avrahami: Absolutely. I do think you need to ask that. You know, there is a lot of trust when it comes to cloud service providers, and that's really a meaningful portion of the isolation that they provide to your workloads. I think you probably would need to consult some security professionals to evaluate the solution that they use. But generally, I think it's something that people should be more aware of. Definitely when I'm doing penetration testing of cloud service providers, this is one of the things I look into, so I'm sure that, you know, malicious attackers are also looking into that.

Dave Bittner: That's Yuval Avrahami from Palo Alto Networks. The research is "Escaping Virtualized Containers." We'll have a link in the show notes.

Dave Bittner: Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Elliott Peltzman, Puru Prakash, Stefan Vaziri, Kelsea Bond, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening.