The CVE countdown clock.
[ Music ]
Dave Bittner: Hello everyone and welcome to the CyberWire's "Research Saturday". I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems, and protecting ourselves in our rapidly-evolving cyberspace. Thanks for joining us. [ Music ]
Bob Rudis: Within the past two years, we've had this what I will call anecdata or gut calls that, wow, we get this weird spike on some piece of usually enterprise-grade edge technology. And then, like, it'll stick in your head that there was a spike and then wait, wait, there's a really interesting or bad or just new set of CVEs that come out like really short period of time later that are either bad or do require patching by folks out there. And, like, that's happened time and time again.
Dave Bittner: That's Bob Rudis, VP of Data Science at GreyNoise. The research we're discussing today is titled "Early Warning Signals: When attacker behavior precedes new vulnerabilities". [ Music ]
Bob Rudis: So earlier this year, we started to say that in our blog posts. Not every blog post, but, like, when there was a decent uptick, either in just normal scanning activity or for a particular CVE, we were like, Huh, that's interesting, that we kept tracking. Six weeks later, roughly between, like, four to six weeks later, we would see another CVE come out, sometimes a really bad CVE come out as well. And then we decided to just, okay, we're making a claim and we're hedging it like crazy in the blog posts. And, like, we need to validate it. Like, we needed to science this thing up. So we took a lot of our data that, so we have an entire new sensor fleet. We have an entire new architecture. So we basically took the entirety of that new architecture, which has been up since about September of last year, and just combed through all of the events that happened there, identified all of the, like, significant spike outliers. And I won't do math on here, but we put the equations that we used inside the report. And took it to, took a look at those spikes and then looked at all of the related hardware, software, whatever technology that was associated with that spike. So, if it was, like, as an example, if it was an Ivanti CVE that we saw a spike on or Ivanti scanning we saw a spike on, we grabbed all the CVEs for the related Ivanti gear that that might have been related to. And then we just looked to see when was the published date of the CVEs after that? And we had a couple hundred spike events across, like, about six or eight technologies, six really well-defined, eight, two more loosely defined, that we're like, Hey, there's a pattern here. Between four to six weeks, you're looking at a new CVE coming out. And we're like, This is useful for us to gauge and to look out for, but we also wanted to have, to make a report about it because it's my thing having been on the defender side and on the vendor side for much, I've been on the defender side a lot longe. It's any leg up you can get on knowing when you might have to prepare your defenses for an attacker is just great, because it's really expensive to, like, set up extra logging or do a bunch of stuff like that. So we wanted to show some documented evidence, show some correlation, because we're not, we don't say causation in the report because it would take a lot more data and a lot more time and a lot more evidence to say real causation. And then after that, it's like, Here, do with this what you will, but we're going to try to put this as a thing that we can put on our product to tell you when there are spikes for a particular technology. But anybody out there with your own logs, you can do the same thing. Because, like, most everybody will see similar activity that we're seeing for some of the opportunistic scanning. And if you have the wherewithal, the bandwidth, the team resources to follow what we did, you could begin to do this predictive stuff too and maybe buy some time with your budget or with your team to prepare your defenses for what might be coming down the pike.
Dave Bittner: Well, you mentioned that it's typically a six week or so cycle. Can you walk us through that? I mean, what happens when these attackers start probing a previously unknown vulnerability?
Bob Rudis: Yeah, so actually that's the really cool thing about this. So, a lot of people are asking us, like, just from within our data, like, you have some older vulnerabilities. Like, we have some 20s, we have vulnerabilities going all the way back but for these enterprise technologies, like, there are some 2017vulns in there, some 2020 vulns in there, and a lot of folks go, like, are attackers actually being successful with those vulnerabilities? And sure, I guess if they get lucky they might be able to compromise a host that's on the internet now, if it really hasn't been patched in that long. I know patching is not perfect at orgs, but generally speaking, a 2017 vuln is not going to be in too many enterprise perimeters. So when they're doing the scanning, like, our belief was, or, like, the CVE-based scanning, like actually testing to see if the exploit works against the particular piece of technology, they're doing that as a cover. Because what they used to do, like, four or five, so, like, back when I was at Rapid7 and I was talking with Morris, like, when he started GreyNoise, we both noticed the pattern of, like, opportunistic scanning going crazy. There were inventory scans happening all the time. And then as both, like, Rapid and GreyNoise, we started to, you know, tell people about this opportunistic scanning. They kind of went silent for a while and, like, they used other stealthier techniques to gather inventory what's on the internet. But they still need to keep an inventory. So our assertion has been, Well, they're doing this probing against older CVEs because orgs don't care about the older CVEs. Like, I hear that from customers all the time, We don't care about old CVEs. So they're getting past their scanning defenses, they're ignoring what that signal is on the wire to just make sure, Oh yeah, there is an Ivanti there, or there is a Fortinet there or there is a SonicWall there or anything like that. And then they use that inventory to then launch new attacks with a brand-new CVE or an 0-day or an n-day that might happen down the pike. So that, our hypothesis was, again, we don't attack people so we don't know if exactly that's what the attackers are doing, but it just seems to be that, like, they have a sense that there is a CVE coming or they're doing their own, you know, exploit creation and they're testing against old CVEs to get under the wire, like, to get under the radar of organizations, and then they use that to have a more precise inventory because they really want to be able to get as many hosts as possible.
Dave Bittner: Well, how often did a spike in malicious activity lead to an actual CVE, and what's that time frame look like?
Bob Rudis: Yeah, so if you look at the report, and I'm just going to scroll real quick just to, like, so I can give you some more, like, some more concrete information.
Dave Bittner: Yeah.
Bob Rudis: So within there, what it looks, if you look at it, and this is, like, page, for folks that are listening and you have the report in front of you, it's, like, page six of the report. There's a, it shows kind of the sequence of spike to CVE. And roughly on there, what you can see is there's 200-ish events. I like to round because people don't like precise numbers. And those 200 events are indicative of that within six-week timeframe. And, you know, it's a handful of CVEs so it's like, I think it's under 30 CVEs total for the whole thing. So that for all across those technologies, there's at least one to three CVEs that came within that six-week time period that basically means the attackers were doing that scanning and knowing in advance, from what we can tell, that this new CVE was coming or that they were the ones that were creating the exploit that created the CVE.
Dave Bittner: And what kind of technologies show the most reliable warning signals?
Bob Rudis: Yeah, so that's a really important question too for orgs. So we looked at every single technology that we have in the edge category. And what we first did is we said, Okay, we need to see, you know, what do the patterns of scanning and attacking look like for these technologies? So I'll give you one that we didn't put in the report and that we didn't analyze. It's, like, D-Link, and actually, I'll throw Linksys in there too. So D-Link and Linksys are extremely noisy, like, technologies from an attacker perspective. Like, they're doing constant probing. It's almost like a heartbeat. So in the report, I think we actually used that terminology. We said, Look, if there's a heartbeat, if it looks like attackers are just going after a particular technology a whole lot, or if there are just scads of CVEs so that we couldn't find any signal in that noise, we ignored that. So D-Link and Linksys and a bunch of other ones that fall into that category we basically threw away, because there was no real spike activity and there was just too much of noise to say that there's signal here. Which ended up leaving us with enterprise technologies, which I was actually surprised at. I actually thought maybe, like, there wouldn't, like, I didn't actually realize there was that many CVEs for D-Link or Linksys or those things. I tracked this stuff, but my gosh, there's a lot of vulnerabilities in this kit. And so for these enterprise technologies, we ended up, like, looking across all of the edge enterprise technologies that are out there. And we didn't deliberately land on, you know, Cisco, Fortinet, Juniper, Palo, I'll list the ones that have more signal in a second. Like, we didn't deliberately pick those, they just happened to be the ones that bubbled up from the signal analysis that we did. So Cisco kit has a, like a, I would say it's an 80% confidence signal that you're going to see that pattern happen on there. But for, like, Fortinet and for Juniper and for Palo Alto and for SonicWall and Ivanti, those ones show a very high signal with this spike to CVE within six weeks. One that I really, I hate to say it this way because, like, I just don't like Citrix, but I was hoping we would see it very concretely with Citrix, and we just didn't see that firm of a signal with Citrix. And a technology that I used to really not like a long time ago, but MikroTik is getting a little bit better in their safety protocols. I thought we'd see a lot more signal in MikroTik, but honestly, attackers aren't even going after MikroTik as often as they used to before. So we really didn't see a whole lot of signal there as well, too. So again, it's, you know, the big players, Ivanti, Fortinet,. Cisco, about 80% correlation, and then Citrix and MikroTik, not a great correlation. Like, I wouldn't be using their signal if I was a defender, you know, and I had to spin up more defenses or spin up more logging. I wouldn't choose to do that if I saw a spike on those. [ Music ]
Dave Bittner: We'll be right back. [ Music ] Can you help me understand the 'why' here? Like, why are these attackers behaving in detectable ways weeks before a CVE is published?
Bob Rudis: Well, so again, like, it is detectable if you know what to look for. And, like, most organizations are so overwhelmed with data hitting their perimeter, whether it's legitimate stuff from applications or whatever they host, or attackers, you know, or scanners. There's just so much volume hitting an organization's perimeter, they have to throw away a bunch of data that they don't care about. So they will ignore, you know, that normal probes, like, lightweight probes or hits against CVEs that they know they've had. Like, they will just ignore that activity because they feel like they've been able to defend against it. And honestly, they have to deal with all sorts of other things right now, that they just can't prioritize that. And so attackers, it also looks like a couple things are happening with attackers. One, they either are the ones creating the exploits that they know they want to deploy, or they're following bug bounty hunters or other research teams, or even wok going on at a vendor. I think if people just take a look a couple weeks ago, I believe it was like it was revealed that Microsoft sort of revealed to Chinese attackers that there was vulns in SharePoint, which caused the SharePoint problem, like, you know, just a couple weeks ago. So, like, vendors aren't necessarily as tight with their information as they should be. Bug bounty hunters aren't either. The bug bounty platforms might not be. Or the attackers are just, you know, creating their own exploits for an eventual 0-day which then will get a CVE almost immediately after the vendors or the researchers figure out what to do. And the scanning that they do, it's really for an augmentation reason or they don't have resources, and I'll explain what I mean by that. So we saw starting around three years ago at GreyNoise that the, what we used to call inventory scans, like, basically you could count on, like, within this couple days a month you'd see this botnet do an inventory scan of the internet for this technology. And after enough of, like, enough orgs like GreyNoise started reporting on that and helping organizations defend against that, that inventory scanning activity went substantially down. And what they do instead is they will take any one of them, like, there's so many internet scanning companies out there right now so I don't want to mention any one of them because it's not their fault that attackers do this. But you take any one of the benign, good guy internet scanners out there, they do that scanning for the organization so the organizations can know what's on their perimeter. Anybody can subscribe and get an account, though. So, if you want to get one of, any one of those providers, you have Shodan, Census, or whatever, and just say, Hey, show me where all the Fortinets are, like, it'll do that, because that's what the whole purpose of that database is. And so, like, most of the time, the, like, the, so we, we divide, we divide attackers into, like, three categories, A-listers, B-listers and C-listers. So the A-listers are, like, nation states and the C-listers are, like, the people that just scrape the bottom of the barrel, spend some time on the Mirai botnet and do what they will out there. So the A-listers stopped doing inventory scans. They just completely stopped doing them. But every so often, they do need to, like, augment the inventory that they have with Census with something a little more recent or a Census showed and however, again, I'm not picking at any one of them because it's not their fault. And they'll use that data for targeted scans, but if they really want to have their own inventory and update it and have a really crisp, like, immediate thing so that they know they can use it within a certain period of time, they need to do their own scans. And they can't do what they used to do, which was just do random scans for technology or fingerprinting or whatever. So, by launching a CVE-based exploit or attempting to do so, making it look like they're idiots, that they don't know what they're doing because why would you try to, like, do that against that piece of technology? It gets under the radar of everybody and that gives them a really fresh, crisp inventory. Now there's also a secondary thing where if an org hasn't patched a 2020 CVE or a 2022 CVE, they can compromise that host as well, too. So, like, they get two potential bangs for their buck. They know that there's a piece of technology that they want there and they may have been able to compromise it at the same time. So, like, they get that win and they don't do them as regularly. So, these spikes don't happen on a regular time period. If you look again back on page six of the report, you know, the, each different graph there has a different time frame. And, like, some of them span between, you know, January of this year to July of this year. Some go back to December or September of last year to now. Some are just more recent. So they don't do the inventory scans with a regularity like they used to do. They do these bursts to try to get under the radar and then, you know, use that data later on for when they want to have, when they know a CVE is going to happen, or they create an exploit that causes a CVE to happen.
Dave Bittner: Well, what are your recommendations then for defenders? What should they do when they spot one of these spikes?
Bob Rudis: Yeah, so if I were back, I thankfully am not in a Cisco position, and I don't have to do what real, what people who have real work for a living have to do, which is, like, defend an org and protect orgs from their stuff, because it's really hard. The one thing that, like, I do know is, like, if you need to do, like, extra logging, like, either from a system log perspective, or let's say that you want to capture a full PCAP or even partial sampled net flow, doing that for, like, all year across all sorts of devices and technologies is you just can't, like, even the fortune 1000, the fortune 100 can't do that for all those kinds of technologies that are out there. So if you see a spike happen against one of these older CVEs, or just, like, one of the scanners because we, a couple of our scanner tags, just the vanilla scanner tags also saw some spikes. If you see that in your logs and you, those, that you should be able to take time for, get that into some big data system and be able to do spike detection. Like, we didn't do, this wasn't rocket science this was basic just core data science statistics that we did on here. It is, you can do that, you'll get that notification and that should be a signal that you should flip on maybe some NetFlow logging, flip on, like, full PCAPs, at least part of the time for different days, or get full system logging going into your expensive Splunk, you know, data warehouse or whatever so that you have that to look for, to see what might be coming against your environment down the road. And that six-week cutoff is actually a pretty decent amount of time to be, you know, to get though that logging set up, to get that extra data capture set up, and be watching for what happens with that extra data capture. And I think if orgs could do that, they would be in a better position to see new, weird activity happening against them, or if a CVE comes out within six weeks, they can also plan with their operations team to make sure, hey, can we make sure we patch that, if a CVE comes out within six weeks, can we make sure that there's the emergency patch procedures ready so we can get those patches in and therefore not let whatever those brand-new, like, one, n-day attacks that are going to happen against those CVE be successful against that organization. So we wrote it really for those two things in mind, to give you the ability to, like, prepare your defenses and logs and just watch for what's happening a lot closer so you don't waste a lot of money to do that and resources. Or on the flip side, coordinate with all your internal teams that you have to because taking, you know, down a Cisco firewall or router, taking down a Fortinet appliance, I can get on the list, you are taking that down, like, there is downtime, you might not have an HA setup, a high availability setup, so if you take that down, it causes real downtime for real people, it's a real business thing. But if you can coordinate it ahead of time, or know in advance you have to watch for these things so you can, like, block IPs on your own if you want to do that, or watch what we have over here, that gives you, like, real power to maybe not be in that first round of attacks that attackers are doing. And again, it does require some resources, it requires, you know, a different frame of mind for the teams. You might actually have to, like, you know, get some folks that know how to do, like, the logging so you can do the time series analysis. But honestly, it's a small investment for them to do that, to potentially save themselves for a, like, really bad attack. Because it just seems lately that, like, CVEs come out and an attacker will, like, launch an 0-day, a CVE will happen right after that, and everyone's taken by surprise, and they just don't have any time to react. We were hoping with this analysis to give them some tools that they can use to be a bit more prepared if you have the ability and time and resources to kind of do that kind of analysis.
Dave Bittner: What about false positives? I mean, were there any surprises where a spike didn't result in a new CVE down the line?
Bob Rudis: That's a wonderful question, and we chose to not put the graph in the report because it was a lot more complex. So there were ones that fell outside of that window, but it was, like, 20, it was like in the 20% range of the spikes that fell outside of the six-week window and didn't fit into there. And they weren't with the technologies that we show in the report. What we chose to put in the report were the technologies where we saw these spikes, there was the propensity of them there. And we included, like I said, the Citrix one and the MikroTik one to show, yeah, like, they have, and, like, and even Cisco, it's only 80% of a signal. So we did show that so you could see, yeah, like, we aren't saying that this is a complete predictor. There isn't, like, solid 100% correlation across all technologies for this, but there's a, there's enough of a correlation for it for certain types of technologies that you can use this to your advantage to be prepared. Like, there's actually one graph in the report I want to see. It's, yeah, so it's on page ten of the report or sorry, page nine of the report. And there's, like, an area graph above what looks like something shooting like little dots out. There's a lot of dots after this. There's not a lot of dots after the six-week line but there are dots after the six-week line which means that that spike to CVE correlation wasn't within that six-week time frame. And that's really important for folks to realize, look at this, you know, choose to set up your defenses if you want to this way, but there is a huge correlation for, again, a good number of technologies that it makes a lot of sense for people to do that.
Dave Bittner: So what are your recommendations for folks who want to get started taking a look at this? What's the best way to check this out?
Bob Rudis: So, most folks do have some type of detection capabilities within their perimeter. I know some of the organizations that just can't afford a whole lot of stuff don't have that, but the ones that you can afford a decent-sized Fortinet or a decent-sized Ivanti or Cisco, like, you do have some capital expense money, like, in your security budget. If they're ignoring older vulnerabilities because you believe you patched them all, I guess the first thing I would say is at least take a look at the CVEs that we noted inside ours, inside our report for those, and have your detections log the hits to those CVEs. And then basically take those things put them in some kind of, it doesn't need to be, like, a gigantic Oracle database, you can get SQL, and, like, they just shove this data into, like, a small database and just regularly run checks against the spikes with the, again, the equations that we provided to you. It literally is that simple, is, like, Hey, log these events, check it for the spike events, and then you do your own comparison. And what's been really interesting about that, actually, I was going to say that earlier. So I had no idea how this was going to, like, we don't pre-release reports to everybody, we just release it to everybody at the same time. And I got so many responses from other researchers and other defenders. So, like, there was, we had one person tell us, Yeah, I was in my DFIR community Slack. I have no idea which one it was. And they were going, we all had this gut call that, like, that we, this felt like it was happening too, but none of us could justify the time to go do that log collection or event collection, the correlation and do that. And a whole bunch of them are now going to go do that with their own stuff because again, this is hitting your perimeter just like it's hitting our stuff. So, like, I would say there is definitely value there. A lot of other folks have said, Wow, yeah, this tracks, like, we felt this too, and I'm glad someone did some data behind it. And my hope is a lot more folks do take that time and kind of double check our work. Like I basically, like, see what, you know, do what we've done against your stuff. If you can go back in time and do that, if you were logging stuff before, that'd be amazing too, because I'd like to know that the correlation, the signal is even stronger than we've already shown for the data that we have. And we're going to be continuing to update this and revisit it over time. So, we can tell people, Yeah, like, the signal is still there, the correlation is still there. Although by doing what we've done, we may have told the attackers that we know what they're doing and they may stop what they're doing, I have no idea. It's one of those things, like, the minute you tell somebody that, like, you see this, they might stop doing what, it's really frustrating when they do that, really frustrating.
Dave Bittner: At the risk of being cute, you know, it reminds me of, I think it was an old Bob Seger lyric, you know, "I saw the lightning and waited on the thunder".
Bob Rudis: Yep.
Dave Bittner: The spikes are the lightning and the CVE is the thunder.
Bob Rudis: Wow, I wish you could, wow, that would have been amazing for the report. That would have been so amazing [laughter].
Dave Bittner: Well, feel free to use it if you ever do a, you know, a TED Talk on it or something like that, right?
Bob Rudis: Oh man, oh man, I don't know.
Dave Bittner: Well Bob, I think I have everything I need for our story here. Is there anything I missed? Anything I haven't asked you that you think it's important to share?
Bob Rudis: I think maybe the one thing I would just say is I know organizations have to be focused on, like, the core defender operations that they have to go do, but they're sitting on a treasure trove of data that if they could allocate just some time to go ask questions of their own data, like, I work for a vendor, so, like, this vendor's not going to be happy I say, well, actually, I think they'll be fine with it. You have your own treasure trove of data, and it's, like, the tools to do data science, just basic stuff on that data, exists, like, all over. They're all free to, like, the vast majority are free. And if you can justify or, like,2 find some way to justify carving out some time to just start asking questions of your data. Like, for all the folks that are listening to this that had that same gut call, you thought, Yeah, this feels like this is what happens, and you've had that thought for the same couple of years that we've had, go talk to your boss and say, Hey, can we carve out, like, three hours a week to go do this kind of stuff on our own data? And I think you're all going to be surprised at what you'll find if you could actually go do that, and, you know, not relying on anybody else except yourself that kind of information. I think you can develop some pretty powerful in-house techniques and in-house tools to repeat this and find more and new and novel stuff that people aren't looking for right now.
Dave Bittner: And it also strikes me that, you know, in this age of AI and automation, that there is still a really important place for that gut feeling, to chase down that gut feeling.
Bob Rudis: Oh, 100%. Like, so the one thing that's really important is you're always supposed to start any type of analysis or process or whatever, like, when doing this kind of work, with a hypothesis. And the only way you're going to get a hypothesis is if you're thinking about stuff. And generally speaking in security, everything starts with a gut call because you're just, you have this Spidey sense that this is happening and this is happening, I need to now somehow determine whether my hypothesis is true, and most people just don't have the time to actually act upon that. And again, I would just urge folks, if you have that gut call, find some way to get someone to give you the time because the resources are pretty much free, and go do that. Because I think you'll find a lot more than you're finding right now. [ Music ]
Dave Bittner: Our thanks to Bob Rudis from GreyNoise for joining us. The research is titled "Early Warning Signals: When attacker behavior precedes new vulnerabilities". We'll have a link in the show notes. And that's "Research Saturday" brought to you by N2K CyberWire. We'd love to hear from you. We're conducting our annual audience survey to learn more about our listeners. We're collecting your insights through the end of August. There's a link in the show notes. Please do check it out. This episode was produced by Liz Stokes. We're mixed by Elliott Peltzman and Tre Hester. Our executive producer is Jennifer Eiben. Peter Kilpe is our publisher and I'm Dave Bittner. Thanks for listening. We'll see you back here next time. [ Music ]
