Control Loop: The OT Cybersecurity Podcast 4.5.23
Ep 22 | 4.5.23

The challenges of carrying out vulnerability management.


Dave Bittner: It's April 5, 2023, and you're listening to Control Loop. In today's OT Cybersecurity briefing, the Vulcan Papers, the Cyberspace Solarium Commission recommends that CISA set up a test bed to improve maritime cybersecurity. Dragos CEO on critical infrastructure cybersecurity. The JCDC's pre-ransomware notification efforts. Today's guest Mike Hoffman is Technical Leader for Global Services at Dragos & a SANS instructor Mike will be discussing challenges carrying out vulnerability management. In the Learning Lab, Dragos' VP for Product & Industry Market Strategy Mark Urban concludes his two-part discussion about industrial cyber threat intel & collective intelligence with Seth Lacy whose Principal Threat Hunter at Dragos.

Dave Bittner: NTC Vulcan, a Moscow based IT consultancy has been exposed as a major contractor to all three of the principle Russian intelligence services, the GRU, the SVR, and the FSB. Vulcan's specialty is the development of tools for cyber attack. Der Spiegel, one of a group of media outlets that broke the story, sources it to a major leak, stating, "This is all chronicled in 1000 secret documents that include 5299 pages full of project plans, instructions, and internal emails from Vulcan from the years 2016 to 2021. Despite being all in Russian and extremely technical in nature, they provide unique insight into the depths of Russian cyber warfare plans. In a militarized country that doesn't just fight with war planes, tanks, and artillery, but with hackers and software. The Vulcan papers reveal that the company is engaged in supporting a full range of offensive cyber operations. Its services and products extend to espionage, disinformation, and disruptive attacks intended to sabotage infrastructure. And the company also provides training to its customers in the security and intelligence organs. The U.S. Cyberspace Solarium Commission 2.0 has published a report calling for the cybersecurity and infrastructure security agency to set up a maritime equipment test bed to enhance maritime cybersecurity, FedScoop reports. The report states the program can begin by testing for cybersecurity vulnerabilities in foreign manufactured cranes used in U.S. ports, as mandated by the National Defense Authorization Act of the fiscal year 2023. And then, expand into broader, systematically important maritime OT. Drago's CEO Robert M. Lee on March 23rd testified before the Senate Committee on Energy and Natural Resources to discuss cybersecurity vulnerabilities in the United States energy infrastructure. Lee first pointed out that the ICS threat landscape shifted irreversibly last year, due to the emergence of Pipedream, a malware framework capable of launching repeatable attacks across the OT ICS industry. Lee stated that Pipedream initially targeted energy assets, but can work in almost all OT environments, including military weapons systems. Lee then discussed how the government should focus on efforts that have been successful and avoid duplicating resources or guidance, stating, "We need to regulate towards outcomes, not prescriptive requirements, using the expertise of the private sector, and be sure they're not counterproductive to what we're trying to accomplish. Such as overlapping reporting requirements that cause confusion." Finally, Lee said the government should identify its critical assets, decide which risks to defend against, and allocate the necessary resources to address those risks. He stated, "The government must be resourced appropriately to protect its own networks. DOE and CISA both require authorities and resourcing to hold the DOE and government agencies accountable for cybersecurity requirements on new projects, such as distributed energy resources. It is difficult for the government to talk credibly on the topic of cybersecurity when its institutions sometimes have less security than most energy sites." The U.S. cybersecurity and infrastructure security agency's joint cyber defense collaborative is cultivating its pre-ransomeware notification capability. JCDC stated, "With pre-ransomware notifications, organizations can receive early warning and potentially evict threat actors before they can encrypt and hold critical data and systems for ransom. The JCDC is a public private sector information sharing organization established by CISA in 2021. JCDC Associate Director Clayton Romans explained in a blog post that pre-ransomware notifications are possible due to tips from the cybersecurity research community, infrastructure providers, and cyberthreat intelligence companies about potential early stage ransomware activity. Romans added that since the start of 2023, we've notified over 60 entities across the energy, healthcare, water and wastewater, education, and other sectors about potential pre-ransomware intrusions, and we've confirmed that many of them identified and remediated the intrusion before encryption or exfiltration occurred.

Dave Bittner: Mike Hoffman is Technical Leader for Global Service at Dragos and a SANS instructor. Today, he discusses challenges carrying vulnerability management.

Mike Hoffman: When I think about vulnerability management, especially in the OT side, you know, a lot of times, we often consider, you know, vulnerability as something, some weakness in a system. And a lot of times, we like to classify vulnerability as something that's the same throughout the environment. And the thing is, though, when you consider OT sites and systems, a vulnerability in one system, depending upon how the system is used and so forth, may not be the same throughout the entire environment. So, it's very much, you know, we have to really take that step back and understand how systems are actually being used, where they're being used at. To understand a vulnerability a little more holistically, I think context is often extremely key when we think about this. One vulnerability is not the same as the other, and one system and how its used is not the same as the other.

Dave Bittner: So, is this a matter of, you know, the same vulnerability in a system that's keeping track of how many snacks there are in a vending machine has a different potential impact than the same vulnerability with a system that's keeping an eye on a turbine? 

Mike Hoffman: Yeah, exactly. And that's one of the challenges, I think. So, if we take a step back from this a little bit, when we consider vulnerability management, oftentimes, we think about vulnerabilities as, it's a patch. It's something that some researchers found some, you know, company has discovered this vulnerability to a piece of software or hardware or even firmware. And oftentimes, you know, we like to, again, treat these as the same. So, normally, when people think about vulnerabilities, we're all used to patching our phones, patching our computers. I mean, everybody's familiar with this idea of keeping our systems up to date. Latest and greatest, you know, features and so forth. The challenges, though, is when we think about this and when we apply that back and we pull it back down to OT. Those systems sometimes can be patched. Sometimes they can't. also, when we talk about the particularities, a lot of times, when we think about, you know, we need to go, you know, patch this Windows server out there, that server, depending upon how it's used, depending upon how the actual application is being used, can have no impact at all. It can have a significant impact. Lot of times, when we think about some of our systems, they may be used or remote access on a Windows server, or it could be for a file transfer solution. Some of those systems, you could patch every month. Wouldn't be a big deal. But that same device or that same VM that's running in the OT space may actually be running your, part of your, you know, DCS software. That one may be a little more challenging to patch. So, when we think about these vulnerabilities, they could be the same vulnerability across the machine, but we can't take the idea that we're going to go out and fix these the exact same way, the exact same frequency, and so forth. We have to take that nuanced approach.

Dave Bittner: And how do you recommend that folks come at this, in terms of prioritizing what they do come at?

Mike Hoffman: Yeah. It's a great question. One of the best questions out there is how do you deal with this. First of all, you have to know it, right? So, a lot of times, when we take a step back, we consider vulnerabilities, it's all about knowing what you have first. And there's a strong argument out there around, well, you know, before we do vulnerability management, you have to have an inventory of your systems and what's out there. And so, you can inventory it a number of different ways, but once you get that inventory and you find this vulnerability, it's like, then what? You know, the struggle is do we, you know, do we correct it now? Do we somehow, you know, do we mitigate by patching? Do we mitigate by other means? Do we have other compensating controls out there? And which system do we touch first? One of the things that I like to, when I'm in a classroom setting or if I'm in a, you know, working with customers, is that I always think about what systems are interconnected with other systems. And in the OT space, this thing about, you know, we've had, for a number of years, we've been interconnecting and bringing together the OT and the IT space very, very tightly, I would say. You look at a lot of industries out there, you'll have, you know, you have recipe movement, you'll have custody transfer measurements, you'll have environmental measurements. All kinds of things. You talked about your little snack machine. Even to do that, in a lot of manufacturing environments, you have recipes coming down or orders coming down on how much of widget x to make. That is all interconnected, and those systems that are transferring data and information back and forth to the IT space are the ones that are actually the most exposed. So, this idea of exposure is very, very important when you think about vulnerabilities. What systems could be impacted first? What systems are more vulnerable to external business side or even cloud connected systems? Those are the systems that I would want to consider first and patch first and/or mitigate first. After that, when you think about the space, the further you go down in the networks, there's already compensating measure in place there or should be. And if not, we need to look at other things that we can do to detect certain type of activities.

Dave Bittner: I'm wondering, I mean, do you have a certain amount of, you know, sympathy for folks who look at their situation and they say, "better safe than sorry?" You know, going back to that snack machine, you know. I'm going to patch that anyway because it is connected to the greater whole, and I'd rather take care of it and know that it's done that not. Do you find that folks often approach things with that kind of a mindset?

Mike Hoffman: Absolutely. And this kind of goes back to especially when you have, you know, companies that, with their OT teams, you know, coming together. They sometimes even being, you know, driven from the IT side. The IT perspective is always patch often and patch fast. When a vulnerability hits, you know, the first thing they do is they go scan their environments from maybe a vulnerability scanner using endpoint agents. Scan quickly access where they're at and patch it as fast as you can. Taking that same mindset in the OT space, it's kind of like you said, it's better safe than sorry. Well, we should just patch as fast as we can or, you know, maybe the risk of unpatching, you know, it's something that we don't know. So, you know, a patch is something that we can do. And so, I actually did this in a prior role that I was at, working at a plant. Is there's so much pressure at that time to go patch our systems. And it felt like that's all I did for, you know, 70 to 80% of my time was out patching all the time. And the problem with that is sometimes, there's so much focus on keeping our systems patched and so forth, that first of all, there's only so much time in a day. There's so many resources. And is that really the most effective use of your resources to constantly patch? Also, there's oftentimes, and this is actually, you know, being seen in, you know, like Dragos year end review reports and so forth, that not all patches correct the actual root cause solution. So, we may be thinking that we're correcting something when we're actually not. We're not fully correcting something. And then, the patch can break something. So, a number of years ago, and this has actually happened multiple different times. Little communication protocol called OPC that relies on DCOM. Microsoft has historically been fixing a lot of some vulnerabilities within DCOM, but the problem is folks have, in the past, implemented this maybe incorrectly or insecurely. And so, as these patches are, have been applied, DCOM breaks. That means your interconnectivity between different systems break. A lot of planned disruptions and so forth. So, patching has a great chance of breaking a lot of our older ICS applications. And so, we have to be very, very careful about this mindset that, yeah, we might be fixing this small security problem, but we may be creating a huge impact to our plant production and so forth. So, there's always a give and take here. And going after patching all the time is maybe not the most effective way to spend our time and resources. It's not a bad thing. We need to be doing it. But it's yet, it's not the fix all solution.

Dave Bittner: Taking this sort of approach, this risk based approach, is this, how does this align with the existing culture in the OT space? Does it mesh, or is it counter? Does it fit somewhere in between?

Mike Hoffman: I think it does. When we think about, sites are used to managing risk, and that's what they do day in and day out. When you consider most of our industrial sites, finery chemical plant, you know, high voltage transmission distribution and so forth, generation. What we are used to dealing with very, very risky environments. The nature of critical infrastructure. And so, when we think about, you know, patching, everything that we do is, should be from that risk perspective that what are we really trying to mitigate? Are, you know, is this, is there another way? So, I think, you know, thinking about a risk based, you know, way of coming about it. Anytime you go out to a site, and you talk about this is a risk that is, we're trying to mitigate. This is a risk that, you know, we could be causing, the harm we could be impacting the site. A lot of times, you know, especially when I've done this before, again, I lived, you know, this patching issue many, many years. Is that you have to take different, you know, you have to take different decisions on some systems we can autopatch that are very, very low risk. If that system goes off line, I don't care. Somebody may, you know, it may cause somebody a bad day, an engineer, but it's not going to affect the plant. But the minute I begin to go to different systems that will have a higher effect, that's when we really need to bring in other groups. You need to talk to your operators. You need to talk to other staff to understand the risk that is there. Again, always test before you do something. Always make sure you have a backup before you apply the patch. But still, that sometimes, you have to take that decision that we're not going to do this today. We're not going to do this next month. We may have to wait until five years, until this part of the plant is down for maintenance until I can actually apply this firmware or software patch. And so, everything should be taken through the lens of that risk, that understanding of the risk of it, what you're actually mitigating, and so forth. So, absolutely this resonates with folks at sites.

Dave Bittner: In today's Learning Lab, Drago's VP of Product and Industry Market Strategy, Mark Urban, concludes his two-part discussion about industrial cyber threat intel and collective intelligence with Seth Lacy, who is a Principal Threat Hunter at Dragos.

Mark Urban: Hi, this is Mark Urban with another episode of the Learning Lab on Control Loop. And today, we're going to talk a little bit about the collective intelligence, meaning when different organizations are able to pool anonymized information between themselves to increase the effectiveness. And I'm joined for this topic by Seth Lacy. Seth, could you introduce yourself, what you do at Dragos, please?

Seth Lacy: Yeah. So, at Dragos, I'm a Principal Threat Hunter, which means I kind of tack the adversarial groups that are focused on industrial networks and trying to both penetrate, and in some cases, disrupt those networks.

Mark Urban: Neighborhood keeper. That's what we're talking about in the Dragos context. What makes it unique? So, certainly the no trust approach to it. What other aspects of Neighborhood Keeper are unique to the system?

Seth Lacy: Yeah. So, I think another interesting thing that that no trust approach allows is that trusted advisor part that I discussed, which I think is a really important and unique aspect of the program because, you know, it allows the participation of these trusted advisors that, if the information was more detailed, if identities weren't stripped down to the data, might not, you know, be as possible for them to participate, right? And so, by having their participation, the ISACs, the CERT, some government partners, it allows the advisors to disseminate information on threats, trends, and maybe research they've conducted to the broader Neighborhood Keeper community. But the other thing that it provides is an avenue for participants to submit targeted, kind of encrypted requests for assistance to advisors or other participants in a time of need, right? So, you know, if a participant finds themselves in a bad situation and they say, "Hey, you know, like, I need some help." They have an avenue to reach out and ask for that help from some of these trusted advisors or, you know, from other participants in the program. All while maintaining that level of anonymity kind of right up to that point of needing to ask for help or aid.

Mark Urban: Right. Because if you're asking for help, people are going to want to know a little bit more about you before they respond to that, right. I have 

Seth Lacy: Yes. Absolutely.

Mark Urban: And I think the other thing, it should be within the context of this particular show. It might be apparent, but I'll just call out. This is also focused on industrial security, right, industrial control systems, operational technology, and the unique information base and intelligence that is kind of shared and detected in that context, rather than on the information technology side. Is that, it's an industrial focus, right?

Seth Lacy: Yeah. Absolutely. And that's, you know, that's one of the other, you know, really key points of Neighborhood Keeper is that it does have that OT Network focus, right, and that's, you know, that's something that sets it apart, again, from a lot of the other sharing programs is, you know, Neighborhood Keeper does have that focus on specifically the OT network segment in these organizations, right. It's not interested in monitoring or collecting anything on, you know, what's going on on the IT side of the network. You know, at Dragos, we are really focused on adversaries that want to target OT networks and want to, in a lot of cases, disrupt operations. And so, in the same way, Neighborhood Keeper focuses on that OT network segment.

Mark Urban: Right. So, the industrial control systems, the operational technology and electrical utilities, oil, and gas, pipelines, refineries, extraction sites, and water systems and wastewater systems, in manufacturing, in, you know, whether it's chemical, pharma, or automotive, or, you know, as well as in, you know, we also see operational technology deployed in building automation and building management systems. And it's just a very unique environment, you know, compared to or contrasted to the more kind of broadly resourced information technology. So, it's just, that's just my quick, 60 second bring up of the industrial side versus the IT side that Neighborhood Keeper is focused on. So, if you look at, so if Neighborhood Keeper is about information sharing, it's about kind of sharing the telemetry. It's about being able to ask for help. What makes Neighborhood Keeper a useful tool for network defenders?

Seth Lacy: Yeah. So, I mean, you know, a lot of what it comes back to is, again, talking about the data availability, right. What is your scope of visibility and understanding of what's going on in the community? So, you know, one of the amazing things that Neighborhood Keeper can provide is a certain amount of context for what network defenders are observing within their own OT environment, right. So, if your network defender. You've got Dragos platform. You see a type of detection fire. You suddenly have a place you can go if you're not familiar with that detection and are not really sure how to interpret it, you have a place you can go to get a little bit more context, right. Is this something that's being seen across the wider community, or is this detection specific to my environment at this point in time? And so, you know, I think that level of context is one of the main things that Neighborhood Keeper can provide, but, you know, again, kind of bringing it back to where we started with threat hunting and trying to move to a more proactive approach to network defense. You know, some of these use cases in terms of context are, you know, again, reactive, right. They're a little bit more passive. And so, you know, something we've been looking a lot about, or looking a lot at and actually publishing some blogs on is how to, you know, use the Neighborhood Keeper data more proactively, right. How to integrate some of the context and information available within Neighborhood Keeper to inform threat hunts and generating these hypotheses. You know, working to preempt the adversaries before they can have that disruptive impact, and you know, a lot of this is part and parcel to moving up the maturity curve for an OT cybersecurity program for OT network defense.

Mark Urban: Okay. So, you mentioned passive and active, kind of, use cases. Can you expand on this a little bit?

Seth Lacy: Yeah. So, you know, a passive use case would be like, hey, you know, I'm, I've seen something, fire in my platform instance, and I'm going to, I don't really know how to interpret this detection. So, I'm going to take a look in the Neighborhood Keeper data and see, hey, is this something that is a really common detection in a lot of environments. Or is this detection relatively unique, right? Is it mapped to specific adversary TTPs or is this maybe more like a dual use potentially capability like port scanning or seeing SMP or RDP in the environment that might have a really benign explanation. But again, you're essentially taking that detection and reacting to it, right? One thing we're trying to encourage and kind of, you know, give participants some food for thought in terms of how to use the data actively is, you know, how do I get in front of the adversary, right? How do I get from reacting to the adversary in my environment to instead maybe trying to preempt the adversary, right? So, an example of this might be, hey, you know, I'm, instead of taking a detection that I've seen within my own environment, I'm going to kind of proactively look at what's going on in Neighborhood Keeper, right. And so, as I look at kind of what's trending across the broader community within the Neighborhood Keeper data, perhaps there is a trending detection that's associated with a specific adversary, and it's a lateral movement technique, right? So, I'm going to, you know, maybe take that knowledge and start to try to build a hypothesis, right. So, maybe I look back at some of the previous CTI information I have on the tactics that that adversary uses and I say, okay, so what are maybe some tactics that that adversary uses for initial access, right? My hypothesis is that these lateral movement signatures or detections are trending within Neighborhood Keeper because this is part of an adversary campaign. That adversary is already active, right, in other networks within the community. And I haven't seen them within my environment yet, but the likelihood is there. The possibility is there. So, I want to really understand in the past what that adversary has done for initial access and start developing some hypotheses based on my knowledge of my network, my network topology. How that actor might try to gain initial access to the OT segment on my network. And so, I'm going to go look at things like engineering work stations, jump boxes, and see if I can find that adversary before they're trying to move laterally within the OT network. Maybe when they're just gaining initial access to that OT network segment and hopefully preempt them before they have an opportunity to gain a foothold or much less, you know, have some kind of impact or disruption to operations.

Mark Urban: So, passive is seeing something in your own environment and reacting to that and leveraging Neighborhood Keeper to kind of see if there's a prevalence out in the world. And if you change to active, then it's like, hey, let me go look at what's happening in my broader community. Let me take some indications from there. Now, let me go look and scrub my own environment and more actively kind of hunt within there. Two good, interesting kind of use cases for the technology on kind of the defender side of it. If you look through the lens of threat intelligence generally and threat hunting specifically, what was the motivation of building this, really? 

Seth Lacy: Yeah, you know, I think, you know, the motivation for Neighborhood Keeper was just to build this environment where data could be shared in a way, particularly on specifically the OT network segment that just hasn't been possible in the past, right, because of those privacy and regulatory concerns. And you know, as we've developed Neighborhood Keeper and thought about it, you know, one of our main goals is to be a partner on the journey, right, and to help organizations, you know, mature their OT cybersecurity program as a whole. And so, this is, in our mind, kind of part and parcel to that, right. Is how do we integrate Neighborhood Keeper into that journey and demonstrate ways that the data within Neighborhood Keeper can help you move along. 

Mark Urban: Got you. So, that's the benefit to the participants is just get the benefit of that. Where do you see Neighborhood Keeper going in the future, really? Are there any, you know, what are the goals of where this evolves to?

Seth Lacy: Yeah, so, you know, another aspect of this is having participants use this data in an active way. You know, being in the neighborhood portal, digging through all the information that's available there. You know, just like a neighborhood in the real world, right, or any community, having a more active participant base just makes the community stronger, right. And so, you know, I think the goal moving forward is obviously to continue to grow the size of the community, the size of the participant pool. You know, as they say, knowledge and data are power, and so, you know, the broader that participant base is, then the stronger the community as a whole can be. Right? And so, with things like the trusted advisors and, you know, some of these more proactive approaches to defense and using Neighborhood Keeper data, the more people we have participating, the more people we have engaged, the more powerful that community of collective defense becomes.

Mark Urban: Seth Lacy, ladies and gentlemen. Seth, a Threat Hunter focused on industrial here at Dragos and talking about not only threat hunting and threat intelligence but collective defense with Neighborhood Keeper. Seth, thanks very much for enlightening us about this today.

Seth Lacy: Yeah. Thanks for the opportunity. I really enjoyed talking about it.

Dave Bittner: And that's Control Loop, brought to you by the CyberWire and powered by Dragos. For links to all of today's stories, check out our show notes at Sound design for this show is done by Elliott Peltzman with mixing by Trey Hester. Our Senior Producer is Jennifer Eiben. Our Dragos producers are Joann Rosh and Mark Urban. Our Executive Editor is Peter Kilpe, and I'm Dave Bittner. Thanks for listening. We'll see you back here next time.