Keeping an eye on RDS vulnerabilities.

Dave Bittner: Hello, everyone, and welcome to the CyberWire's "Research Saturday." I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Gafnit Amiga: It actually was for fun - the exploitation part. But the main purpose of the research was to deeply understand how things are working to better detect it in the platform. 

Dave Bittner: That's Gafnit Amiga, Director of Security Research at Lightspin. The research we're discussing today is titled "AWS RDS Vulnerability Leads to AWS Internal Service Credentials." 

Dave Bittner: Well, for folks who may not be familiar with Amazon Elastic Kubernetes Service and all of the stuff that you're working with here, can you give us a little bit of the background, a little description of how all this stuff works? 

Gafnit Amiga: Yeah. So Kubernetes is the containers orchestration platform that was created inside Google. EKS is a managed Kubernetes service. It's really complicated and hard to maintain a Kubernetes cluster, even just to create it. It's - it can be difficult sometimes. So what Amazon are doing is to give us, like, a black box - all the control plan area. They create the cluster for us. They maintain the control plan. And you can have access to the data plan, and you can just start and connect your client to the cluster and start deploying - deployments on Kubernetes. It should be easier to work with that and maintain it, and EKS is that managed service. 

Dave Bittner: Well, that all sounds great. Walk us through your research here and how you discovered some vulnerabilities. 

Gafnit Amiga: In EKS, the research question was first, like, how EKS performs the authentication because they are able to identify you as an AWS IAM identity. In AWS, you have users, you have roles, you have groups. And in EKS, you can stay logged in to the terminal as an AWS identity - let's say, an IAM user or role - and you can perform actions against your managed Kubernetes cluster and EKS. So the question was how it's done and to deeply understand it. In Noga's research, you can see exactly how they are doing that. She specified that there is a ConfigMap inside the cube system namespace. And this ConfigMap - the aws-auth ConfigMap - there is a mapping between AWS identities and Kubernetes identities. And the code that is running in their control plan by AWS is the code that if - of the AWS authenticator for Kubernetes. This is an open-source code that was developed first by Heptio and then moved to AWS EKS themselves. And they're using this code and deploy this service inside the managed Kubernetes control plan. You cannot see it. You cannot access it. But it's there - or, like, some kind of a version of it. 

Gafnit Amiga: And the logic there is written in that code. And by, like, researching the code and doing a code review, you can see what is the exact logic of how things are being translated between the AWS identity and the Kubernetes identity. And when I did the review and the debug, I locally installed it and just to play with it - to be able to play with it. I saw that there are multiple parameters in the request that they are looking in order to decide what is going to be that Kubernetes identity they are going to map to. They want to figure out, eventually, what is going to be the Kubernetes username and Kubernetes group. And in order to do that, they're using AWS service and endpoint, which is STS get-caller-identity. And AWS STS get-caller-identity returns back the logged-in user identity. So they use this endpoint in order to get back the user identity, and then they take this identity and map it using the ConfigMap to a Kubernetes username and group. So in the way they are using the STS get-caller-identity endpoint is that they're requesting the user to access the cluster to send a signed request to STS get-caller-identity endpoint. And the signed request is assigned using the AWS credentials, the access key and the secret of the IAM user in AWS. When you go over the code, there is a section when they take the signed request part. So there are multiple parameters part - that are being passed as part of this request. But one of the parameters is, for example, what is the access key ID of the user that signed the request? 

Gafnit Amiga: So there are multiple parameters being passed in the request. And they are taking these parameters and using a ToLower function in order to save it. So there is a kind of a trick that you can pass the same parameter name-wise. And then you sign the request with both parameters. But one is, like, with uppercase, and one is with lowercase parameter name. And what happens is that the EKS cluster will take one of the two - because it depends on the ordering - but it will take one of the two. So we want the cluster to take the malicious one, the one that we're going to manipulate the value of the access key ID. And the STS in AWS will take only the one that he is familiar with. He will not - it will just ignore - the STS service itself will ignore the manipulated parameter as long as it's signed as part of the request. 

Gafnit Amiga: So this is exactly what I did. I passed both the - I passed twice the same parameter name in the request and signed it. And then what happened is that EKS took the malicious access key ID and translated it into, like, a different user. So you can trick it to think you are another identity in the cluster. This can lead to a privilege escalation inside the cluster. It depends on how the mapping in the aws-auth ConfigMap is configured. But yeah, this is the technical issue. So it's a - it's led to a privilege escalation inside the cluster. 

Dave Bittner: And has the EKS team addressed this? 

Gafnit Amiga: So they - what they did is to check that you are not passing the same parameter name twice. They allow you to pass it only once, and then it's - it prevents the attack. 

Dave Bittner: I see. So is this fixed now? Has this been patched? 

Gafnit Amiga: Yes, it has been. It was patched, and it was fixed. Actually, it was patched very quickly. And now where all the EKS managed clusters are fixed, but if you're using the open-source of the IAM authenticator or that you're using EKS Anywhere, which is the on-premise version of EKS, then you should take an action to update this service by yourself. 

Dave Bittner: What do you make of this vulnerability - in terms of, you know, how this code was written and the functionality that they were intending to have happen and the resulting bug? I mean, do you have thoughts on how this sort of thing comes to pass? 

Gafnit Amiga: Well, I think that the service is written really good. You can see from the - in the issues, you can see that they're - in the pull request, you can see that they're taking into account all the security aspect. They have blocked many other leads that I thought that I will be able to exploit in the code there. They heavily thought about security while they created this extension. It's tricky. It's always tricky to enforce the connection between two services - and in this case is Kubernetes authentication and identity and AWS identity. It's always tricky but I guess the need to ease the usage of the end user, at the end. But I think that here, also, when you look the way they treat it - so they took it very fast from the moment you report to the moment it's deployed in production to fix. So I think it's a good balance here. 

Dave Bittner: You know, you mentioned when we were first speaking that part of your reason for digging into this was fun. Was it fun? I mean, it sounds to me like this is the sort of thing that you and your colleagues really have a good time with. 

Gafnit Amiga: Yeah, yeah. It's always interesting because you start just by researching to understand how it works, but then things start to come up to your mind, like - and what if? And if I will change that into that, how it will react? And you always save it to a later time. And then when you have the time, you try to play with it. So it is - for me, it's fun. 

Dave Bittner: Our thanks to Gafnit Amiga from Lightspin for joining us. The research is titled "AWS RDS Vulnerability Leads to AWS Internal Service Credentials." We'll have a link in the show notes. 

Dave Bittner: The CyberWire podcast is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Rachel Gelfand, Liz Irvin, Elliott Peltzman, Tre Hester, Brandon Karpf, Eliana White, Puru Prakash, Justin Sabie, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening. We'll see you back here next week.