It was only a matter of time.

Dave Bittner: Hello everyone, and welcome to the CyberWire's Research Saturday, presented by Juniper Networks. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Dave Bittner: And now a quick word about our sponsor, Juniper Networks. NSS Labs gave Juniper its highest rating of "Recommended" in its recent Data Center Security Gateway Test. To get your copy of the NSS Labs report, visit juniper.net/securedc, or connect with Juniper on Twitter or Facebook. That's juniper.net/securedc. And we thank Juniper for making it possible to bring you Research Saturday.

Dave Bittner: Thanks to our sponsor, Enveil, whose revolutionary ZeroReveal solution protects data while it's being used or processed – the holy grail of data encryption. Enveil delivers privacy-preserving capabilities to enable critical business functions. Organizations can securely derive insights, crossmatch, and search third-party data assets without ever revealing the contents of the interaction or compromising the ownership of the underlying data. What was once only theoretical is now possible with Enveil. Learn more at enveil.com.

Larry Cashdollar: You know, one of my co-workers had come to me and said, hey, we're hearing about this SaltStack vulnerability that's going around and there are systems being compromised on the Internet – do you know anything about it? And I'm like, well, no, I've actually never heard of SaltStack before.

Dave Bittner: That's Larry Cashdollar. He's a Senior Security Response Engineer at Akamai Technologies. The research we're discussing today is titled, "SaltStack Vulnerabilities Actively Exploited in the Wild."

Larry Cashdollar: So, the first thing was to go and start looking at what SaltStack was. I hadn't used it before. And it turns out that it's a – it's like a management system for other systems.

Dave Bittner: Hmm.

Larry Cashdollar: It's like an infrastructure automation software where you can use it to, you know, manage your systems, push configurations, update services, things like that, to your collection of systems on your network. And, you know, I hadn't ever used it or heard of it. So, the next thing was to try and collect information about what was going on. I had a friend who had his system compromised send me some of the binaries that he had found. He sent me some of the scripts, that I used to collect some of the other binaries and scripts that he didn't see on his system. And then I began investigating, you know, what systems were being targeted, what were they being targeted with, how were they being – you know, was there persistence being established? What were the perpetrators actually doing on the system? And then I was pointed to a Slack channel where there was a group of SaltStack users and researchers that were investigating the same thing I was. So I jumped on there and started, you know, getting information from them, sharing information and just trying to help them decipher what the malware was doing. And we were sharing samples with each other, sort of like a collaborative community thing.

Larry Cashdollar: And then I issued an – I sent out a blog post just to alert the Internet as a whole, you know, Akamai customers as the whole, as to what was going on and that you were indeed running SaltStack, that you needed to patch it immediately and possibly investigate, make sure that, you know, your system hadn't been compromised already, because there were widespread attacks that had already been published in the news, and there were already some major sites that had been compromised. So, an active attack, and trying to just help people who were affected get their systems cleaned up and patched, and people that were unaware, to make them aware that, hey, you know, this vulnerability's actively being exploited on the Internet, and if you're running this, you need to patch it immediately.

Dave Bittner: Well, can you describe to us what the vulnerability is and how it was being exploited?

Larry Cashdollar: So, the vulnerability was disclosed by researchers at F-Secure. There were two vulnerabilities. One was remote code execution and the other is a – I think it was a path traversal. I had zeroed in on the remote code execution because that was the most interesting one. So, what this would do was it would bypass authentication and allow a malicious user to execute code or – yeah, code on the SaltStack system as root. And that was pretty severe. That's the most severe vulnerability you can really have with the system, is just unauthenticated code execution as, you know, a systems-administration-level user. 

Larry Cashdollar: So, this is the type of vulnerability that attackers are really looking for, so it was only a matter of time – a short amount of time, in this case – where they started abusing it. And because I believe that the SaltStack systems themselves are probably relatively beefy systems – they're probably multicore CPUs, use plenty of disk and RAM – these systems were targeted for cryptomining software. So, a piece of malware that we saw initially was called "salt-storer," and what it would do is it would kill off other possible pieces of malware, but it would also kill off any sort of services that we're using resources or resource-intensive services on that system and then begin mining malware. It would install a copy of XMR miner and begin mining cryptocurrency.

Larry Cashdollar: So, that was what these guys were mostly fighting. It had a remote access tool where it would set itself up in cron and pull down a script from a remote server and then execute that script every minute in cron to ensure you don't have some sort of persistence on the system. And then if there were any commands being issued, they could be pulled through this HTTP request to this script that was being executed run from a remote server. There were, last I knew, five different versions of this, each version more malicious than the next. So, this – you know, this piece of malware actually went through, like, a development cycle. Now, initially there wasn't any persistence, and then the developer of this malware created some sort of persistence using cron. And then, you know, it was actually killing off every single possible piece of Linux malware, so they kept adding different things to look on the system for and kill. Eventually it was flushing iptables firewall rules with "iptables -F". So, clearly the person who developed this was developing and testing and making adjustments of their malware as they went along. So, it was an interesting thing to investigate and work on.

Dave Bittner: Yeah, it's an interesting thing you mentioned here, that I think is a little bit of side information, but is worth mentioning – that scans revealed that about six thousand deployments of Salt were exposed to the Internet, and that shouldn't be?

Larry Cashdollar: Yeah, typically the SaltStack stuff was usually on an internal network, and I think perhaps folks had installed it and then accidentally left it exposed to the Internet because it really should be something that is kept on an internal network and firewalled off from the rest of the Internet.

Dave Bittner: Hmm. So, in addition to patching, perhaps checking that is probably a good idea as well.

Larry Cashdollar: Yeah. You know, make sure that your services are firewalled off from folks that shouldn't be accessing them or using them. And, you know, if you've got something that's not supposed to be on the DMZ, you really should make sure that it's not on the DMZ. Some of these services seem to be not as hardened as services that are normally offered out on the Internet as a regular thing.

Dave Bittner: And so far, it's primarily been cryptomining that you've seen?

Larry Cashdollar: Yeah, that's really what I've been seeing. It's all been cryptomining malware. Hopefully folks have gotten their systems fixed up and patched so that they can get ahead of this before other pieces of malware start to be developed and added to the list of things that they're installing on vulnerable SaltStack systems.

Dave Bittner: Is it fair to say, I mean, if they can execute code and they have root access, then I suppose the sky's the limit on the possibilities of what they could do?

Larry Cashdollar: Yeah, that's correct. If they've got root, then they can destroy the system, exfiltrate data. Actually, I think the last thing that I had heard was they were beginning to start to exfiltrate data off of these systems. So, they were copying files off. So, it's really, really something that you need to get ahead of and patch and firewall off and make sure that your system's secure if you're running SaltStack.

Dave Bittner: And so, the bottom line here, I suppose, is patching, right?

Larry Cashdollar: Yep. Patching, you know, keeping an eye on your vendor, vendors notifying their customers when a new patch comes out. And then, you know, especially working with vulnerability disclosures, vendors should really work with security researchers and folks that disclose vulnerabilities and then create a timeline of when the patches will be ready, notifying your customers of those patches, and then when that vulnerability is going to be publicly announced. Giving your customers a chance to patch systems before an exploit is widely being used on the Internet for it is a plus.

Dave Bittner: Yeah, it's interesting to note also that one of the victims of this noted that the cryptocurrency mining had spiked their CPUs, which is one of the things that alerted them that they had an issue.

Larry Cashdollar: Yeah, cryptomining is very CPU intensive, so if folks are noticing resources on their systems spiking, specifically CPU that they don't normally expect – you know, if you're like, hey, my backups normally run at 1:00 AM, why is my CPU spiking at 8:00 PM? – it might be something to investigate.

Dave Bittner: Our thanks to Larry Cashdollar for joining us. The research is titled, "SaltStack Vulnerabilities Actively Exploited in the Wild." We'll have a link in the show notes.

Dave Bittner: Thanks to Juniper Networks for sponsoring our show. You can learn more at juniper.net/security, or connect with them on Twitter or Facebook.

Dave Bittner: And thanks to Enveil for their sponsorship. You can find out how they're closing the last gap in data security at enveil.com.

Dave Bittner: The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team working from home is Elliott Peltzman, Puru Prakash, Stefan Vaziri, Kelsea Bond, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening.