A fine pearl gone rusty.

Dave Bittner: Hello, everyone, and welcome to the CyberWire's "Research Saturday." I'm Dave Bittner, and this is our weekly conversation with researchers and analysts, tracking down the threats and vulnerabilities, solving some of the hard problems and protecting ourselves in a rapidly-evolving cyberspace. Thanks for joining us. [ Music ]

Coby Abrams: So Rusty Pearl is actually just from the languages that the vulnerability uses. Right, the vulnerability is in two language extensions in Postgres, and those language extensions are Rust and Perl. So if you're -- if you're imagining, you can either imagine like, you know, a Rusty Pearl, kind of like think Pirates of the Caribbean, or you can imagine the language extensions, the different syntaxes and whatnot.

Dave Bittner: Our guests today are Tal Peleg, Senior Product Manager, and Coby Abrams, Cyber Security Researcher at Varonis. The research we're discussing is titled Rusty Pearl - Remote Code Execution in Postgres Instances. [ Music ] Tal, could you walk us through what remote code execution vulnerabilities mean when it comes to Postgres and what that means for organizations?

Tal Peleg: Sure, so a Postgres database usually, you can execute queries on them. Sometimes you have a web application or some other thing that's running a query on the data that you put in your database, but that database is hosted on some system. And actually what a remote code execution, in this case, what it let us do is actually run commands on the underlying system.

Dave Bittner: So let's go through the just -- the journey of discovery here. What led you down this path of exploration?

Tal Peleg: So I was looking at a ticketing service for just understanding how to secure it on our platform and also trying to look for vulnerabilities, since these are less-explored areas. Most people, most companies, look at maybe file storage or virtual machines, containers, workloads. And we thought, well, ticketing systems, they have all the stuff that's happening inside the company. People are sending information to one another, maybe about customers, maybe about -- maybe these contain sensitive data. Maybe a customer is asking a question, and their question contains some sort of sensitive data. So this place is a treasure trove of sensitive information. And so I was looking at one of these ticketing systems, and I found a SQL injection. And it was a pretty, pretty major SQL injection. And I thought, well, what would happen if I was an attacker? What impact could I have done? And so I set up a Postgres instance on AWS RDS, which is where this -- where this ticketing system was hosting its own database. And I just wanted to see what impact I could do with just a database. And so what I noticed is that AWS provides us with an administrative user, but not a super user. So a super user in a database is able to do -- is able to control the entire database, change settings, even run commands on the system, but this user is not a super user, and they cannot run commands on the system. They cannot open sockets. They can't access files on the system, apart from very, very specific files. And so what I was trying to do is find a way to escape that and either escalate privileges to a -- to a super user or in some other way from commands on the system.

Dave Bittner: Well, take us down that path. What -- how did you go at this question that you'd asked yourself?

Tal Peleg: So I started by looking at extensions, and extensions are a place that are sometimes not tested as heavily. So the database itself is open source. Lots of people have been looking at it. It exists since the -- I don't know since when it exists for a long time. Postgres is one of the more well-known databases, and hopefully that means it's pretty secure, but third-party extensions can be less rigorously tested, not tested in all the same environments that the database was tested in. Maybe interactions between different extensions, since they're usually not tested together, all the different combinations that you can make. So I was looking at extensions. One extension that had a vulnerability for accessing files is the logging extension, something that I noticed, but someone already disclosed it before I was able to. But I was -- I continued looking at extensions, and I was thinking, okay, functions, triggers, maybe something that I can cause the super easier to run in order to bring me higher privileges. So I was looking at functions. I was looking at extensions. I thought, well, there are language extensions. And then one that caught my eye is Perl since Perl exists in the Postgres repo, so it's an official language; it's also open source. It probably exists in lots of Postgres installations, and so it would be pretty impactful to find something there. And also, Perl is from the '80s, so there's a lot of code written in Perl that's before security was really -- or code security was really popular. People didn't really think about all the about vulnerabilities and privacy and data security. So I was looking at Perl, and Perl has an interface to manage environment variables. And managing environment variables is one of the things that only a super user should be able to do. So I thought, well, maybe this interface, which is a magic environment -- magic variable in the function, would let me modify environment variables. And it turned out to be true. So I was able to set environment variables, and then I was looking for some way I could use those environment variables in order to run code. And I looked specifically at extensions that were allowed in AWS RDS Aurora, or AWS Aurora, and I actually wasn't able to find anything. And I put that aside for a little while, and then -- and then I got Coby up on the team.

Dave Bittner: Coby, do you want to jump in here and describe your contributions?

Coby Abrams: So when Tal told me that he has a -- he has this primitive, I was I was really excited. It sounded really cool, but Postgres kind of makes it difficult to execute code using environment variables because, as an extension, when you -- when you're trying -- when you create a new process as an extension in Postgres, you actually have an API in the Postgres it gives you to create that process, and you're not creating it as a child of your own. So it's not taking those environment variables that you've edited in Perl and then creating the process with them, which makes it difficult because the processes that you're now creating have their own environment variables that aren't edited. So what we looked for was an extension that creates processes directly without using this API, right? So it's doing something that it's not really supposed to do. And what we found was that PL/Rust, which is a fairly new extension, actually creates processes directly when it's compiling the Rust function, so that was pretty cool. And once we were able to find that, we found -- so basically, we found that there was a little bit of validation. There is a denialist that denies certain environment variables from being edited while Rust is compiling, while PL/Rust the extension, is compiling your function, and we were able to find an environment variable that wasn't in that deny list that was able to be used to execute code. So it was pretty cool, especially seeing it work when we were -- I was kind of skeptical that we were going to be able to at this point, and suddenly you see the command running, so it's pretty fun.

Dave Bittner: Yeah, I suspect, I mean, that must -- for both of you, that must be a gratifying, kind of aha moment.

Tal Peleg: Right, we even took it a little bit far. So we were try -- we saw we were able to execute code on our own Postgres instance in a container, and so we thought, well, let's see if this thing works on the cloud, on AWS. [ Music ]

Dave Bittner: We'll be right back. [ Music ]

Coby Abrams: So we did -- we did want to -- want to check this out on AWS. We were kind of wondering. We went, and we read all of the different disclaimers that Amazon put out, trying to figure out if like we were allowed to, at all, and it was a bit of a gray area, but we decided to go for it anyways. And of course, we did it on Thursday, like just before the weekend, kind of in the evening, and ended up causing some issues because we were unavailable after that, and AWS were trying to contact us, and they're getting in touch with our bosses, and they're trying to -- trying to call us. They sent -- they actually sent me a LinkedIn message, like they were very adamant on finding out who exactly was running code on their on their RDS instances. So that was --

Dave Bittner: So I guess the good news is AWS detected it. The bad news for you all is that they knew it was you, right?

Tal Peleg: Well, it can be bad, but it's also pretty good because actually - 

Dave Bittner: Yeah. 

Tal Peleg: -- got a nice collaboration out of it. So they were -- they were being very responsive. They were -- they thought we were researchers since Varonis is security company, and we do do security research on AWS, and we do protect our customers' AWS environments. So they know what we do, and they saw that the account was called something with the word research in it, but still running code. It was actually a holiday or just before a holiday, and it was right before the weekend, and so they were a little bit unsure of whether it was a legitimate research or maybe someone compromising the environment. 

Dave Bittner: Yeah, that's interesting. I mean, everybody in security gets, I think, a little on edge when we come up on a long holiday weekend, right? History has taught us to be extra vigilant.

Tal Peleg: Yes, for sure.

Dave Bittner: So what did -- what was AWS's response here? You had this positive interaction with them. What ultimately did they do to protect their managed databases?

Coby Abrams: So within a very short amount of time from us running code, they were able to block our access completely. They put the database in a -- in a -- in a stage, in a mode that we couldn't run -- continue running code. They also reached out to us to us to make sure that we were actually researchers. So they were really good. And once, you know, we got in touch with them and contacted them, they helped us because it's a third-party vulnerability, right? It's not really a vulnerability with AWS. It's more Postgres vulnerability. They were also really helpful in contacting the Postgres team and making sure that things were fixed quickly, and patching it on their servers, asking us to test it and really just being really responsible about it and helping us to make sure that, you know, everybody's data stays safer, which is all of ours -- all of our goals at the end of the day.

Tal Peleg: Right, and after we talked about this in Vegas a few weeks ago, we also were able to actually meet the team that responded to this incident. So that was pretty cool too.

Dave Bittner: Oh, that's great. I mean, you two did present at DefCon this year, so it sounds like it was really a positive experience all around.

Coby Abrams: Yeah, so we were able to present at BSides Las Vegas and at DefCon --

Dave Bittner: Okay.

Coby Abrams: -- which was really nice. You know, that experience of speaking at DefCon and, you know, getting on the big stage was really exciting for the both of us. We were a little nervous as first-time speakers, I think. Luckily, it was kind of the end of the day, and it was a little more low key, like not such a -- such a big talk, but I think it was really great and really exciting, and it was an interesting experience. I really look forward to being able to do it again.

Dave Bittner: Well, congratulations to you both. I'm curious, you know, in terms of solutions here and recommendations for people who are using Postgres, what sort of practical steps should they be taking, I guess, to lock down these extensions? Is that what we're talking about here? 

Tal Peleg: So first of all, I'd say upgrade at least the minor versions of databases. I know it's a little bit difficult, but it's always a good idea. Use roles, so Postgres, but also other databases have role-based access controls. Use those to limit who can create extensions, who can access which databases, which tables, and everything. And apart from this, Postgres also has a configuration variable allowed extensions that specified which extensions you can install on the database. So if an extension doesn't exist there, you won't be able to install it if you use that variable. So I would recommend using that variable and only using extensions that you know that you're using.

Dave Bittner: Anything to add, Coby?

Coby Abrams: Yeah, I think -- I think there's also something to be said for the cloud providers side of things in this -- in this case. Right, we're not just running code on any database, although this vulnerability works in most Postgres databases that have the two extensions installed. But when it's -- when you're running code on a cloud provider's backend, right, there are various things that you might be able to do. You might be able to access credentials that shouldn't be accessed. You might be able to have network access to different tenants. And I think it's really important. This isn't the first time that a vulnerability that allows executing code on database backends has come out, and especially when they can be run on cloud providers, we need to be able to have research, like community research, on those backends to figure out if there are credentials that shouldn't be there, if there are network access that shouldn't be allowed, if there are container escapes that we can implement there. So I think the cloud provider also has a really large amount of -- so the cloud provider has a lot of attack surface there.

Dave Bittner: So, Tal, when it comes to specifically dealing with the cloud, are there any specific messages you want our listeners to take away from this?

Tal Peleg: All right, so in the cloud, managed databases are managed by the cloud provider, but the data is managed by the cloud customer. So there is a shared responsibility here, and sometimes that's confusing, both to the customer and maybe to the cloud provider, and so you need to be extra careful in how you manage your data in the cloud. So while problems -- issues like this one, like a vulnerability in the database itself, is something that the cloud provider should manage, since the extension is something that the cloud provider provides and doesn't allow the customer to manage themselves, the customer should still be sure to use the proper security controls, not let just anyone access their database, so not -- keep their database -- keep their databases private, use role-based access controls, like I said before. Just in general, store the right data in the right places. Use maybe rules to mask some of that data and so on. Another thing that exists in the cloud is cloud credentials. So maybe you want your database to access other data stores that are stored in the cloud. Maybe you want other services to be able to access the database, and those integrations should also be managed carefully because if you have cloud credentials inside of your database, suddenly a simple SQL injection can turn into a complete cloud compromise and lateral movement even between accounts.

Dave Bittner: Well, Coby, can we touch on the research itself and the lessons that you took away from actually going through this process?

Coby Abrams: Yeah, sure. So one of the biggest lessons I think we learned along the way is that setting your goals when you're researching is really important because otherwise things can get really, really hectic when you're researching. Like there are all kinds of different primitives that you can find and attack surfaces that you can look at. And if you just set your goal, like when we found the -- when Tal found the SQL injection, at some point, he decided, that's it, I'm going to find remote code execution on Postgres and specifically RDS or Aurora, and you know, just that goal allows you to be able to kind of weed through the things that are less important and to really focus and have that -- have that goal in mind to be able to succeed in the end, so I think that's one of the biggest lessons we've learned. Of course, there are other things, like environment variables that are fun to use and things like that, but the main -- the main goal is set your goals before you start researching. Of course, things might change along the way, but having them there is a good -- a good idea, I think. [ Music ]

Dave Bittner: Our thanks to Tal Peleg and Coby Abrams from Varonis for joining us. The research is titled Rusty Pearl - Remote Code Execution in Postgres Instances. We'll have a link in the show notes. And that's "Research Saturday" brought to you by N2K CyberWire. We'd love to know what you think of this podcast. Your feedback ensures we deliver the insights that keep you a step ahead in the rapidly changing world of cybersecurity. If you like our show, please share a rating and review in your favorite podcast app. Please also fill out the survey in the show notes or send an email to CyberWire at n2k.com. This episode was produced by Liz Stokes. We're mixed by Elliott Peltzman and Tré Hester. Our executive producer is Jennifer Eiben. Peter Kilpe is our publisher, and I'm Dave Bittner. Thanks for listening. We'll see you back here next time. [ Music ]