The secrets behind Docker.

Dave Bittner: Hello everyone, and welcome to the CyberWire's "Research Saturday." I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Alon Zahavi: The research started as Linux capabilities research and not Docker. But as I went forward with it, I found that Docker had a vulnerability and Microsoft had one, too. 

Dave Bittner: That's Alon Zahavi. He's a security researcher at CyberArk. The research we're discussing today is titled "How Docker Made Me More Capable and the Host Less Secure." 

Alon Zahavi: With both of those vulnerabilities, I succeeded at escalation of privileges in a Linux machine. 

Dave Bittner: Well, let's start with Linux itself then. Can you describe for us, what are the capabilities here, and how do they lead us into a bit of trouble? 

Alon Zahavi: Traditionally, Linux had two kind of processes - processes that ran with root permissions and processes without root permissions. Since Linux kernel version 2.2, those permissions were divided into units called capabilities. And each capability granted the process the ability to perform certain privileged operations. So let's take, for example, the mount SISQL, OK? So before capabilities, only root could have mount new images to the file system. Right now, in order to mount this image, a process will need to have the CAP_SYS_ADMIN capability. Even if it's not root - it can be any user with any UID - and if it's got this capability, the SYS_ADMIN capability, it can mount every image to any part of the file system. 

Dave Bittner: And so what does that mean for us in terms of this vulnerability that you discovered? 

Alon Zahavi: As I said before, even if you are not root but you have certain capabilities, you can do certain privileged tasks. So in case of the vulnerability, I found if you had a GDB, which is a debugger in Linux, with the capability of SYS_PTRACE, you can attach yourself, as a debugger, to any process, even if it's high-privileged process, if you had that GDB. If you had access to execute this GDB with this capability, you can do practically anything. You can attach yourself to a root process which has, by default, all the capabilities in Linux and just do anything your mind limits you to. 

Dave Bittner: Now, I mean, is this a fundamental flaw in the way that Linux handles this, or is this a particular thing that you've discovered here? 

Alon Zahavi: So Linux capabilities - as I said, it's there from Linux 2.2, which is 1999, OK? It's been there for a long time. But only recently, with the development of cloud and containers, that we see this mechanism started to been used. In case of the things I found, it's been misused by mistake by some vendors. But, yeah, every time process asks to perform privileged tasks, the kernel itself checks if this process has the capabilities to do so. If not, it's just throwing an error and said, you can't do it. Permission error. But if you do have this capability, it just moves on, you know? It check - if you have it, you can do it. If you haven't got it, you can't do it. 

Dave Bittner: So how does this affect Docker containers? 

Alon Zahavi: OK. So in Docker containers, by default, you get limited capabilities, because if you had all the capabilities, if the root user from in the container had all the capabilities, it was also the root outside the container. So you can escape it, like, if there wasn't any container jail, OK? So... 

Dave Bittner: Right. 

Alon Zahavi: ...In this case, it didn't had the ability to escape container. But because of how Docker managed file permissions at the version that I checked, any user, unprivileged user, on the host - not a container, on the host - could have access to files that are saved inside the container. So in theory, an attacker that had both access on the host and the container itself could have done anything he wants. He could have crafted any executable inside the container and give it any capabilities he wants or even setuid capability - and then move to the host, run it from there and become root, become the most powerful user in Linux machine. 

Dave Bittner: Part of what I find really interesting in this research you published is you walk us through the process of how you discovered this. Can you share that story with us? 

Alon Zahavi: Yeah, of course. So this research started as Linux capabilities. I wanted to find misconfiguration - or mis-implementation, to be more exact, inside the Linux kernel in order to bypass those checks, those capabilities checks. But as I went forward with the research, I found that the implementation was kind of straightforward, you know? It's - if I talk in code, OK, it was only a if statement, OK? If you got it, you got it. If not, you don't got it. So then I went to search, who uses this mechanism? And I found that Docker uses it a lot. Then I scanned over 2,500 Docker images from Docker Hub and found that Microsoft released few images that had that GDP file with file capabilities, which gives any user that execute this GDP the capability to attach itself to any process, again, even root processes. And that was the first vulnerability I found in this research. Then in order to execute this GDP, we needed the permissions, you know, classic Linux file permissions. And after looking at the files and directories that Docker engine produce when you pull an image from Docker Hub, I saw that one of the directories where the image is saved allows any user to execute any file from within this directory. And again, then I took the image I downloaded from their website, the Microsoft image. And using low-privileged user on the host, all I had to do is just find the path and then execute the file. From then on, I was root for any - I was root, actually. That's it. 

Dave Bittner: Yeah. Yeah. That's enough, right? 

Alon Zahavi: Yeah, I think that's enough. 

(LAUGHTER) 

Dave Bittner: That's kind of everything (laughter). 

Alon Zahavi: It's like the Holy Grail for Linux attackers, you know? 

Dave Bittner: Right. I'm imagining you. You're sitting there, you know, at your workstation and kind of - you know, your eyes opening up and wondering to yourself, wait a minute, is this really what I think it is? 

Alon Zahavi: It's exactly as you said. I worked. And I tried to execute it. And I was like, no, it shouldn't work. 

Dave Bittner: (Laughter). 

Alon Zahavi: No way. And then I just press enter, and it worked. And I was sky high. I went to my colleagues and said, oh, my God. Oh, my God. I found it. That was what I was searching for. 

Dave Bittner: (Laughter) Right. So is this a matter that - you sort of alluded to this earlier, that, you know, this feature was introduced into Linux back in the very late 90's. Was it kind of sitting around because, you know, back in the '90s, nobody was thinking about cloud environments the way that we do today? 

Alon Zahavi: I think that it was lying around because people didn't want to use it. You know, it messes things up. It's not easy as either you privileged or you not. This kind of mechanism gives you the ability to create the least privilege environment. And, as I said, it messes things up. And when you talk about cloud and containers, you have to do it. When you using a container which has the same kernel as the host, you have to do it. And if you do not do it, the attacker will have, you know, easy attack surface. So that is why, in my opinion, clouds and containers uses Linux capabilities more than the traditional Linux machines. People should have more thinking about this mechanism, so people will use it more. Because if more people will use it in its correct way, the machine will be much more secure. 

Alon Zahavi: Because, for example, take the ping command, OK? It's not like this anymore because they changed it, but it used to be a setuid binary. So every vulnerability you had in a ping, which - all it does is send an ICMP packet, OK? If you had a vulnerability there, you became root. But with capabilities, if you found vulnerability in ping, all you had is, like, you can send more ICMP, you can build a raw socket and that's it. You will become root. You will have more privileges, but you're still not root. You're still not the most powerful user in Linux. 

Dave Bittner: I see. You're limited in what you can do within ping itself. 

Alon Zahavi: Yeah. Yeah. 

Dave Bittner: Yeah. And that's not the case here. 

Alon Zahavi: No. Again, the GDB, which is, again, a debugger with the capability to attach itself to root processes - it gives you the ability to - again, to attach yourself to root processes and change the stack and change the memory and change the code flow - OK? - the running flow. You can do anything. You can do - you can inject shellcodes and just change the register that points to the next instruction and just run this shellcode. So that's why it's so dangerous to have this capability that executable. 

Dave Bittner: So where do we stand now? In terms of Docker itself, they - they've updated this. Where are we? 

Alon Zahavi: Yeah. So the file permissions that I talked about regarding to the Docker vulnerability, it was fixed, I think, few month ago - few months ago. So that's fixed, OK? But that's just one vulnerability. So Microsoft fixed their vulnerability also. But it doesn't mean that this escalation of privileges is not there anymore, OK? So Microsoft and Docker did what they had to do. But it's now the task for the IT teams and the system administrators to update Docker engine and to update the images, and they need to scan their machines, their servers, so they know for a fact that these kind of vulnerabilities are not there anymore. 

Dave Bittner: Is there anything to be done on the Linux level itself of limiting, you know, Linux's ability to do this? 

Alon Zahavi: I don't think so, no, because this mechanism is supposed to make things more secure. And when you use it right, it does make it more secure. It's right for everything. You know, if you misuse something, it will be dangerous. But when you use it correctly, it's great, really. 

Dave Bittner: Our thanks to Alon Zahavi from CyberArk for joining us. The research is titled "How Docker Made Me More Capable and the Host Less Secure." We'll have a link in the show notes. 

Dave Bittner: The CyberWire "Research Saturday" is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Liz Irvin, Elliott Peltzman, Tre Hester, Brandon Karpf, Eliana White, Puru Prakash, Justin Sabie, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe and I'm Dave Bittner. Thanks for listening. We'll see you back here next week.