How are we doing in the industrial sector?

Dave Bittner: Hello everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Sergio Caltagirone: And so, you know, it was really a good opportunity to step back and ask the question, what's changed? But more importantly, also I think that's important is what hasn't changed.

Dave Bittner: That's Sergio Caltagirone. He's Vice President of Threat Intelligence at Dragos. Today, we're discussing their 2020 ICS and OT Cybersecurity Year in Review.

Sergio Caltagirone: We've been doing this report for three years now. Obviously, Dragos has been around for four-and-a-half years. And, you know, really one thing we've always looked at is other vendors and other members of the community who have been able to put together real data points about cybersecurity. And I think that we all, in this space, we all have all of these anecdotes about stuff happening. And that's great, but it doesn't – anecdotes don't make good policy. And I think that's true almost everywhere.

Sergio Caltagirone: And what Dragos really wanted to do was say, hey, look, ICS and operational technology, the systems that run our power, our water, our food manufacturing plants, and they keep us safe and healthy, and they produce drugs and pharmaceutical factories and so forth – like, we need data about that, too. It's not just about email systems being compromised, or web browsers, or zero days affecting Zoom or whatever. It's really about, you know, how are we doing in the industrial sector specifically? Because it is a very unique one. And so, every year now we've done a report, and it's a bear, man. I got to tell you...

Dave Bittner: (Laughs)

Sergio Caltagirone: ...It's one of the hardest things I do, because it takes so much work to really pull apart – you know, after a whole year of work, it's hard to pull apart, like, you know, and step back after you're like, you know, you fight fires every day and you step back and you're like, OK, well, what really happened this last year? Of course, 2020 being unique in that we have all these other global events that have been placed on top of us as well. And so, you know, it was really a good opportunity to step back and ask the question, what's changed? But more importantly, also, I think that's important, is what hasn't changed, and what needs to change. And so, that's why we put this together and we really try to make it data-driven.

Dave Bittner: All right, well, let's use that as our point of departure then. I mean, what are the things that you tracked in terms of what changed and what didn't change?

Sergio Caltagirone: Yeah, you know, and I love to say that our 2020 year-end report is awesome, but more importantly, it's how you put the reports together and also how you look at them in terms of comparison to other reports in the industry to get a better sense of, like I said, the strategic cybersecurity picture for industrial. You know, for us, we have been tracking threats, we've been tracking vulnerabilities, and then we also have been tracking our engagements. So when we show up on-site at a customer environment, the question is, well, what did we find? What didn't we find? What was available to the defenders? What wasn't? What worked and what didn't? Basically to help all of us kind of get better at understanding what's going on in the world, what's happening with our systems in the vulnerability space. And then ultimately when bad things happen, what's going on when we get there?

Sergio Caltagirone: And so really, we break the report down in those three steps, because all three of those steps are necessary for bad stuff to happen. You've got to have a bad person out there doing things. You've got to have some way for them to do it – a vulnerability. And then finally, you have to have defenders who were unable to defend their systems effectively. And so, all three of those are kind of part of the picture of creating a defensible industrial control environment.

Sergio Caltagirone: Fundamentally, so, in the threat space, you know, it's really important to recognize that we've only been looking at industrial threats directly and focused on them for the last four, four-and-a-half, to five years. And what we've found is that over that period of time, the threat landscape is growing. That's an easy thing – everybody in the cybersecurity community says that, by the way, so that statement alone shouldn't cause any issues. But what's really important to realize is that we're growing at about a 3x rate. And so, we are tripling our threats – for every threat that seems to kind of go quiet, we're getting three new ones. And for us, that's really, really important.

Sergio Caltagirone: And we're just sitting at the tip of the iceberg because ten years ago, when Stuxnet came out and everyone was like, whoa, what just happened here? And everyone looked around at everyone else and was like, uh, is this something different? Is this – does this matter? All of the major adversaries in the world started putting money and resources into this problem, like, hey, well, somebody can take down our manufacturing systems, we should be able to take down theirs. And what we found is this 3x shows perfectly that ten years ago people really started investing and it takes five to seven years for an investment in a new attack surface to really bear results.

Sergio Caltagirone: And so what we found is this year we added four new dedicated ICS attack groups to our inventory. And that's really impressive for a single year, especially for one where we're really just looking at the beginnings of this. The second area is vulnerabilities. And this is one where we've really focused on, are vulnerabilities being captured accurately, and do people have the information necessary to understand them? There's thousands of vulnerabilities in the world, Dave. What are we going to – how can an asset owner understand which ones they need to patch today? Obviously, you know, they're like, oh, well, let's look at what people have written about it, you know, the governments or whatever. We found that in seventy percent of the cases, the publicly available information about industrial vulnerabilities underreported the severity of that vulnerability, which means that in most cases, asset owners were not operating on accurate information when it came to which vulnerabilities they needed to address in their environment.

Sergio Caltagirone: In addition to that, thirty percent of the vulnerabilities that were publicly reported in 2020 were wrong for industrial. And that, again, is another point to say vulnerability analysis in the industrial space is very poor. And so there's a lot of work that has to get done so that when we do patch these systems – some of these industrial environments only go down once or twice a year, some only go down once or twice every decade – when they have the chance to really get in there and clean and patch and do all of this, what are the ones they're going to work on? Well, they need good information to decide that.

Sergio Caltagirone: And the last thing that really kind of jumped out at us was that in ninety percent of the cases where a customer calls us and says, hey, there's a bad thing happening, can you come help? Ninety percent of those cases, Dave, there was no data available to help. And that continuously shows – and that's not a change – that continuously shows how this really fundamentally has – our visibility into the industrial landscape has not yet fundamentally evolved. We're getting better, but still, there's too many industrial environments that are having cybersecurity problems and they just can't see them. And of course, the rule is if you can't see it, you can't protect it. And so we really need to get better there. So, I think in all of those three areas, we're showing that it's really getting worse, but there are huge opportunities for us to do small things to get better.

Dave Bittner: Help me understand the sort of – the spectrum between the number of folks who are coming at these systems – you know, the bad guys targeting ICS environments – but also the amount that folks are now looking at them, are looking for those bad guys to be coming. You follow me here? 

Sergio Caltagirone: Yep.

Dave Bittner: Like, how much of this is actually an increase in attacks and how much of it is that we're actually looking for them now?

Sergio Caltagirone: Oh, Dave, you hit – you're going right after my heart here. 

Dave Bittner: (Laughs)

Sergio Caltagirone: So, that's called visibility bias in intelligence analysis, right? 

Dave Bittner: Yeah.

Sergio Caltagirone: So, sometimes you can – well, first of all, I like to say that visibility is king of intelligence, right? Because you can only know what you can see. And so we again, we only know what we can see. And the fact is that the answer is yes, Dragos has changed, for a lot of people, in a way that we are looking specifically for things that people weren't looking for before. So the answer is we are absolutely going to find things that were around. Now, what's important to realize is that we try our hardest to balance that bias with external data sources that can give us an insight as to when this threat began. And so what we try to do is measure not just is there a threat, but what is the earliest timeline of that threat. So that then, yes, we can answer that question better as to when did this start, which is new, or did this – like XENOTIME, one of the threats we track, we have evidence of them going all the way back to 2014 and we found them in twenty seventeen. And so, yes, but in the cases of this year, most of the threats that we're finding are new. From the visibility that we have, we're able to confirm that, yes, these are threats that have just begun in the last year to two years.

Dave Bittner: In terms of how we take action based on the information that you all have gathered here, to what degree are we behind? Is this a Manhattan Project kind of thing where, you know, we've got to get all hands on deck and work on this? Or is there a more deliberate sort of rational, kind of slow thing where we can plan and say, OK, you know, over the next X number of years we are going to get to this point as a nation?

Sergio Caltagirone: Yeah, that's a great question. And I want I want also to recognize that this isn't a US problem, right? That this is this affects seven billion people worldwide who use industrial control systems for reliable power and clean water and so forth. So, you know, this is a global issue. And when attackers attack a system in, say, India and they affect an industrial control system there, they're learning how they attack industrial control systems elsewhere. So you see that very traditional threat proliferation problem. And so that's why we very much treat this as a global issue.

Sergio Caltagirone: I think what we've seen especially with, say, the water treatment facility in Oldsmar, Florida, and with other incidents that happened last year and over the last couple of years, I think we're seeing increased urgency. Four years ago, Dave, when I think you and I first talked, this was very much a, hey, things aren't bad, not bad yet, they're going to get worse, we can kind of see that, you know, we have time. I think that that clock is running out on us. And I think that we're not getting better fast enough. And I think the answer is that we are getting left behind. We had the opportunities, you know, four or five years ago to get better when we knew this was going to be a problem. And I think that we're not yet seeing the amount of acceleration to protect these environments that we should have. And my concern is that this is slowly turning from a, hey, you know, we can do this, it can be methodical, it can be improved, we can get better. And I've got to say, over the next three to four years, this is going to turn into a Manhattan Project.

Sergio Caltagirone: And this is – we are in a very important situation where we know what we need to do. There is no question that water treatment plants need to be protected. The answer is going to be, what do we do about it? And the answer is, it's coming. Right? It's here, and it's going to come even more. It's going to come more often. So the answer is we need, first of all, the answer is visibility, visibility, visibility. I've hit it several times in this podcast so far. If you can't see it, you can't protect it. And so with that ninety percent statistic of most organizations don't even have the basic data to protect themselves, we have to start there. And if we don't start there when we have an Oldsmar, we're going to get stuck in the same situation of something bad happened, but we don't entirely know what or how or when or so forth, and we need to get better at doing that. And that is our first step to understanding the adversaries and then to lay the foundation of greater defensive action as we move forth.

Dave Bittner: Forgive the naivete of my question here and nerding out a little bit, but is there an element where market pressures can contend to outstrip the security realities? And what I'm thinking of is, you know, somebody has a pump sitting out in the middle of nowhere and that pump needs to be monitored. And, you know time was perhaps we sent a couple of folks out on some interval and they went and they took measurements on that pump and made sure that pump was working. Well, now that pump is remotely monitored. You know, the miracle of 5G... 

Sergio Caltagirone: Correct. 

Dave Bittner: ...Has allowed us to remotely monitor that pump, right? 

Sergio Caltagirone: Right.

Dave Bittner: And has the fact that we're not actually sending real human beings out there – we're trusting the data stream that's coming from whatever monitors we put on that pump. Are we – kind of, to mix metaphors, are we outkicking our coverage?

Sergio Caltagirone: Oh, that is a great question, Dave, and it's not naive at all – this is actually wonderful. In fact, this brings me to one of my favorite facts about industrial control systems. One of the reasons why rural telephone networks were legislated to exist inside the United States was because of the need to manage electricity over long distances. And so, in fact, one of the earliest uses of rural telephone lines was to manage remote electrical stations. And so, in the industrial space, we are in no way, shape, or form, you know, afraid of remote or distance issues. In fact, it's pretty much baked into most ways we think about industrial. And the key element is that the adversaries right now, yes – will we potentially have to question our monitoring systems and things and our sensors? Yes, we will probably have to do so. But just like in the land of IT, the adversaries are still such that they don't really care if they got caught. It doesn't really harm them fundamentally. They're doing a little bit of log manipulation and things like that in IT, but not really. You know, it's so funny how much data you get about them when you have the data. And I think, so, for us, it's a great question. But I think that is a question that we should be hopefully addressing in ten years, not this year. That is a future issue. And I think we still have to get to the "can we see it?" before we get to the question of "can we trust the data that we see?"

Dave Bittner: Hmm. Mm-hmm. Well, I mean, big picture take-homes, what do you hope people walk away with after they've read the report?

Sergio Caltagirone: One is that there should be public pressure generally on public policymakers to improve the cybersecurity systems of public utilities. That has to be a critical element of what we do. In addition to that, private entities need to recognize the raw data here and say, OK, if we have a major incident in ninety percent of the cases, we will have no idea what just happened. And that is not OK if you want to be able to bring a plant back up online safely. And so I think both from a market pressure, from downtimes and industrial operations stoppages and so forth, and disruptions, all the way back to the public utilities need to be protected. We need to have reliable and safe electricity and drinking water and so forth. I think we need pressure on both sides to make industrial systems better.

Sergio Caltagirone: And so I think that there's a role for everybody. There's a role for people reading this report and listening to you who are like, I have never touched or I don't even know about industrial systems. Well, you know what, call your Public Utility Commission and say, what are we doing about this? Right? Talk to your legislators. Talk to your local governments. Talk to people who have control over this happening for your communities. You don't want to be an Oldsmar, Florida. And in addition to that, the company leaders who are listening to this need to start looking at the data and say, wow, we have an industrial environment and this is coming at us like a freight train. We should probably do something about it now. So I think there's something in this report for everyone to take away and do something.

Dave Bittner: Our thanks to Sergio Caltagirone from Dragos for joining us. We were discussing their 2020 ICS and OT Cybersecurity Trends Year in Review. We'll have a link in the show notes. 

Dave Bittner: The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Elliott Peltzman, Puru Prakash, Kelsea Bond, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening.