Exploring vulnerabilities of off-the-shelf software.
Dave Bittner: Hello everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.
Tomislav Peričin: And the question was, well, how many of these outdated dependencies, which might be vulnerable to some serious CVEs, are out there?
Dave Bittner: That's Tomislav Peričin from Reversing Labs, where he's Chief Software Architect and co-founder. The research we're discussing today is titled, "Third-party code comes with some baggage."
Dave Bittner: Well, so, walk us through how you approached this. How did you all get started on this research journey?
Tomislav Peričin: Yeah, so we actually scan all these packaged repositories continuously. One of our core missions is to be able to find any malicious packages which might reside in them. So what we do is we scan things like NuGet, and npm, and, you know, PyPI, and all the other open-source repositories. So that drew us to the conclusion of, well, we're checking for malware. What would happen if we start looking for vulnerabilities as well?
Dave Bittner: And how did you go about doing that? I mean, how do you – what's the process for digging in here and finding what's buried in these packages?
Tomislav Peričin: So the package managers themselves host all of the versions of the packages themselves. You can relatively easily go and crawl all of the packages themselves. You end up collecting a vast amount of data. So, npm, I think is about nine-million different package versions that you kind of crawl and maintain over time. And then obviously after you have all these packages, you start doing static analysis. So, unpacking them, analyzing the contents, looking at, you know, are they only using script languages intended by the package manager or are they also including, like, native binaries as well.
Dave Bittner: And what what did you find here? Take us through some of the results.
Tomislav Peričin: So, with that idea, then we said, OK, so we know a lot about analyzing, for example, Windows binaries, and we know we can have static linked libraries in those applications. Is this something the developers are actually actively updating as well? So that led us through this path of, you know, making some signatures to identify commonly used compression, encryption, and networking libraries that we know for a fact are updated all the time, and they're updated all the time because frequently vulnerabilities get discovered in them. So with those sorts of signatures and doing this large data analysis, we were able to uncover, you know, hundreds of thousands of different packages across many different versions they have that actually do still have vulnerabilities which might get exploited.
Dave Bittner: So help me understand here, I mean, is this a matter of of people using other snippets of code and open-source things where, as you said, you know, popular libraries and so on – is it a matter of them using ones that are out of date that and not going through the process of updating them? Is it neglect, or what exactly leads to these being consistently within the things that you're looking at here?
Tomislav Peričin: Yeah, so it's not on the end user. So, the end user uses the package manager correctly. So they say, OK, update all the packages, bring down the latest versions, and the developers of those packages tend to update that code frequently. However, the statically linked libraries do not appear in the list of dependencies, and you have no idea if the original developer has actually updated the statically linked libraries or not. And as our research shows, there's quite a few developers out there who kind of include a third-party project and forget, and then just ends up accumulating this large number of vulnerabilities in open-source repositories.
Dave Bittner: Can you give us an example of some of the things that you found here?
Tomislav Peričin: Yeah, so so we've looked for, as I said, common compression, encryption libraries. And, for example, zlib is the most common compression library out there. There's only like two dozen or so versions that it has, but almost every single version has a vulnerability or two. And it's relatively easy to identify if you create a signature which just includes the banner, which says, I am zlib, a specific version, you'd be able to identify if the library is actually statically linking an outdated zlib version. So we did that, and through the scan we found quite a few popularly used packages which statically link these things.
Tomislav Peričin: I think one of the most interesting one was the application, which is used to kind of edit and create new DICOM images, and DICOM images are used in medical devices, right? So this is very interesting because they're using an outdated version, and then if you're parsing an image which has a vulnerability within, the parsing itself could trigger a code execution. So that's why, you know, having these libraries continuously updated is important, even though they might not be declared by people creating the packages.
Dave Bittner: So, how would someone go about doing what you've done here, I mean, in order to, I guess, it's – is it a matter of how deep do you dig? I mean, if I'm developing something here and I've got these dependencies, what's the best way for me to go about making sure that there aren't these sorts of things lurking within?
Tomislav Peričin: So, for developers is relatively hard because the tools they have at their disposal coming from the package manager themselves, they only show the declared dependencies. To be able to find the ones which are not, you need to use additional tools. And they are commercially available tools, they're called SCA tools, or software composition analysis tools. A lot of them will just look at the package manifest as well, and stop there. The ones which are really good will try to do static analysis and figure out, is there any statically linked code. Those are limited and they tend to use a predefined set of signatures to find common vulnerabilities. So, we would recommend using those in addition to open-source tools.
Dave Bittner: Hmm. In terms of the bad guys being able to exploit this, I mean, if it's not obvious to the developers, is it also not obvious to the bad guys that these vulnerabilities would be here? Or is there a disproportionality here, where the bad guys have the advantage?
Tomislav Peričin: It is relatively simple to create these signatures. So, let's say you're a bad guy and you're trying to do kind of mass exploitation of these vulnerabilities. For the statically linked libraries, which are easy to identify like we did, writing a simple signature will do. You can easily find the vulnerable packages, and you can find them way before the actual developers do. We haven't yet really seen large-scale exploitation like that. But given what's happening with software supply chain and attackers finding new and novel ways to exploit the developers, this might be just a matter of time.
Dave Bittner: And what about the notion of a software bill of materials? I mean, would that is that the sort of thing that would help out here, where if you had a manifest of everything included, would that help us get where we want to be?
Tomislav Peričin: Absolutely. That is the way forward. However, most software build materials for open-source packages are being created based on the self-declared list of dependencies. Additional tooling is actually required to be able to detect statically linked code and add that to the software bill of materials. Typically, software bill of materials is created manually by the developers, and it suffers the same drawbacks as as anything else. You know, maintained by humans, it can be outdated, it might not be accurate, it can change over time, dependencies could be forgotten or added. So it really is up to security providers to actually develop tooling that can accurately define the software bill of materials.
Dave Bittner: And how do you rate the seriousness of this? I mean, to what degree should this be on people's radar? How seriously should they take the possibility of a vulnerability here?
Tomislav Peričin: Well, let's start with the basic one, which is, you know, do I understand the software bill of materials for the packages I'm actually using? And if the answer is no, and in most cases it is going to be no, what can be done about it? Because we already are looking for, you know, software bill of materials to detect certain issues, right? We're trying to find license violations. We're trying to find vulnerabilities in commonly used components. Extending that process to make us a little bit more secure is actually required.
Dave Bittner: So, what are your recommendations then? For folks who want to do a better job here, how do they get started? What's the best way to approach this?
Tomislav Peričin: Well, I would look at additional security tools. If you are not using any software composition analysis tools, that would be a good start, because we've seen that the gaps in protection are actually being exploited by attackers. So we've seen people misusing the vulnerabilities abilities. We've seen people misusing the lack of vulnerability mitigation protections enabled. And adding additional tooling, both commercial and open source, is required to start bridging that gap. And we do see the actual attack surface, and so the packages that we provide to other people, and other developers are depending on are a little bit more secure.
Dave Bittner: Our thanks to Tomislav Peričin from Reversing Labs for joining us. The research is titled, "Third-party code comes with some baggage." We'll have a link in the show notes.
Dave Bittner: The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Tre Hester, Elliott Peltzman, Puru Prakash, Justin Sabie, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening. We'll see you back here next week.