Real world stories of incident response and threat intelligence.

Dave Bittner: It's August 23rd, 2023, and you're listening to "Control Loop." In today's OT Cybersecurity Briefing, radiation sensor reports from Chernobyl may have been manipulated. A South African power generator has been hit by malware. APT31 is linked to attacks on industrial systems in Eastern Europe. Environmental regulation and increased maritime cyber risk. The CISA Director warns of Chinese infrastructure attack staging. Threats to the power grid. CODESYS vulnerabilities. This episode's guest is Dragos' Lesley Carhart, sharing their RSAC 2023 talk on real-world stories of incident response and threat intelligence. The "Learning Lab" continues with the conversation between Dragos' Mark Urban and Kimberly Graham about the convergence.

Dave Bittner: Citing research by Ruben Santamarta, Wired reports that radiation sensor data from the Chernobyl exclusion area may have been manipulated during the Russian Army's brief occupation of Chernobyl during February and March of 2022. The sensors showed troubling, but inexplicable, spikes in radiation levels. Those reports appear to have been bogus, the data possibly manipulated by a cyberattack. The published abstract of Santamarta's talk says, "Evidence confirms that the radiation levels depicted by a very specific set of real-time radiation maps, which during those days were consulted by millions of people and also consumed as a single source of information by media outlets and official entities, did not correspond to the actual physical conditions of the Chernobyl Exclusion Zone." If the data were indeed manipulated in a cyberattack, then that's troubling. Corruption of sensor data in industrial systems would represent a major safety issue for many sectors and for the public at large. NBC News reports that CISA Director Jen Easterly warned at DefCon that Chinese threat actors have been conducting battle space preparation against U.S. critical infrastructure. Easterly stated, "I hope that people are taking seriously a pretty stark warning about the potential for China to use their very formidable capabilities in the event of a conflict in the Taiwan Straits to go after our critical infrastructure." Chinese authorities, of course, deny any such activity and say, "This sort of thing is just disinformation from the Americans," whom the Chinese call "the champions of hacking." Director Easterly's remarks are based on a sound old military custom of planning based on adversary capabilities as opposed to necessarily speculative conclusions about adversary intentions. Those capabilities were on display as recently as this past May. A joint advisory from all Five Eyes reported a major Chinese cyber espionage operation that succeeded in penetrating a range of U.S. critical infrastructure sectors. Microsoft, in its own report on that activity, says, "The group responsible has been active since at least the middle of 2021." The targets of the spying have extended to the communications, manufacturing, utility, transportation, construction, maritime, government, information technology and education sectors. Microsoft writes that "observed behavior suggests that the threat actor intends to perform espionage and maintain access without being detected for as long as possible." "It does this," the Five Eyes stress, "by carefully living off the land, exploiting existing legitimate administrative tools and privileges in its targets." Much of the activity, which Microsoft refers to as "Volt Typhoon," has been directed against Guam, the U.S. territory in the Western Pacific that hosts important U.S. military bases. Those bases would be important to any U.S. intervention on behalf of Taiwan should China decide to take a page from Russia's geopolitical playbook and invade what it regards as a renegade province. For its part, China dismisses the reports as a coordinated American disinformation campaign, and denies that it's engaged in any of the activities the Five Eyes and Microsoft associate with Volt Typhoon. Microsoft assesses that this attack could be an effort to enable a disruption of communications between the U.S. military and its Asian allies. Microsoft stated, "Volt Typhoon has been active since mid-2021, and has targeted critical infrastructure organizations in Guam and elsewhere in the United States." "Security researchers at Microsoft discovered several high-severity vulnerabilities affecting the CODESYS Industrial Automation software," The Record reports. Microsoft worked with CODESYS to develop patches for the vulnerabilities. And organizations are urged to apply the fixes promptly. The flaws could be exploited to carry out denial-of-service attacks or remote code execution. Microsoft notes that exploitation of the vulnerabilities requires user authentication, as well as deep knowledge of the proprietary protocol of CODESYS Version 3 and the structure of the different services that the protocol uses. The vulnerabilities aren't trivial, but they don't seem likely to be open to devastating exploitation either. Ars Technica quotes experts who think it unlikely, for example, that such exploitation might shut down significant fractions of a power grid. Jimmy Wylie and Sam Hanson, researchers at Dragos, point out that CODESYS isn't as widely used in power generation and distribution as it is in other industrial applications. This renders it unlikely that an attack on the software would affect the electrical power sector. Turning to the requirement for authentication a hypothetical attacker may face, Wylie and Hanson observed that if an adversary is authenticated, has the username and password to your PLC, you've got bigger problems than these CVEs, and they can do all kinds of things that make the CVEs unnecessary. They, also, point out that industrial systems are designed with a degree of resilience built into them. So, keep calm and keep patching. Kaspersky warns that a new version of the SystemBC malware was used in an attack against a critical infrastructure power generator in an unnamed South African nation. Kaspersky says, "An unknown actor targeted an electric utility in Southern Africa with Cobalt Strike Beacons and DroxiDat, a new variation of the SystemBC payload. We speculate this incident was in the initial stages of a ransomware attack. This attack occurred in the third and fourth week of March 2023 as part of a small wave of attacks involving both DroxiDat and Cobalt Strike Beacons across the world." DroxiDat, a lean variant of SystemBC serving as a system profiler and simple SOCKS5-capable bot was detected in the electric utility. Kaspersky offered tentative attribution of the incident to a Russian-speaking cybercriminal gang, specifically to FIN12, which has also been called "Pistachio Tempest." FIN12 has previously been known for attacks against the healthcare sector. In May of 2022, it was one of the gangs prominently featured in the U.S. Department of Health and Human Services report "Ransomware Trends in the HPH Sector." FIN12 has changed its target selection, but not its playbook. The group's motivation is financial. Some news reports have said the incident occurred in South Africa, but that hasn't been confirmed. As we note, Kaspersky has said only that it took place in an unidentified country in the southern part of the African continent. In an article for Dark Reading, Jeffrey Wells from Sigma7 outlines the cybersecurity implications that the European Commission's Fit for 55 environmental regulation will have on the maritime industry. Wells notes, "Vessels must now significantly reduce their carbon intensity by making substantial investments in advanced technologies and sophisticated equipment to enhance vessel efficiency. Integrating technologies with existing OT systems and real-time cloud-based monitoring presents a unique challenge to maritime cybersecurity, a field marked by inherent vulnerabilities." Wells concludes that, as a result, "The urgent call is to act boldly and decisively. Immediate investments in cutting-edge technologies, system upgrades, stringent access controls, network segmentation and rigorous vendor vetting are paramount." The Daily Dot reports that a suspected White supremacist in the U.S. threatened to attack power grid infrastructure unless two other alleged neo-Nazis were released from prison. The two imprisoned men were arrested earlier this year for allegedly planning a bank robbery. The individual who made the threat posted four diagrams of power grid equipment on Telegram. The U.S. Department of Homeland Security is monitoring the threat. It's not merely a threat arrived at as a matter of a priori possibility destroying or damaging power distribution substations, as for some time figured as a common topic of fantasy and discussion within this particular extremist subculture.

Dave Bittner: Lesley Carhart is Technical Director for Industrial Incident Response at Dragos. I spoke with them about their RSAC 2023 talk on real-world stories of incident response and threat intelligence. Here's Dragos' Lesley Carhart. Lesley, great to see you again. The show going well for you so far here at our RSA Conference?

Lesley Carhart: It's going fantastic. I've done the hard part of giving my talk. So, now it's easy from here.

Dave Bittner: Well, let's talk about that. You were part of a keynote panel this morning. Can you share some insights? What was that all about?

Lesley Carhart: It was on incident response. And it was a rockstar panel I was very privileged to be a part of with Wendi Whitmore and Katie Nickels and Lily Newman. Just a phenomenal group of people talking about instant response and what's going on in that space and what's in the store for the future in that space.

Dave Bittner: Can you share with us some of the highlights? What were some of the insights that the group shared?

Lesley Carhart: So, we talked a lot about the evolution of threats and how ransomware attacks, and criminal actors are changing their tactics, as well as what state actors are up to these days. But we also talked a lot about the challenges that we face in incident response as a profession. So, that's everything from mental health and burnout to hiring pipelines. It's very challenging to get new people into the field. And, also, things like planning for incident response and how to share information and how to make hard risk decisions about what to do in incidents. So, it was a wide array of important topics that are challenging "it depends" kind of questions.

Dave Bittner: Yeah, I'm really interested, as a leader in that space when, you're looking to bring people onto your team, what are the personality elements that make for a good incident responder?

Lesley Carhart: You have to have a couple different important skillsets. So, first of all, you have to be a good investigator. That doesn't necessarily mean you have to have all the technical skills right away, but you have to have a good investigative methodology and mindset. So, you need to be able to understand the scientific method for building a hypothesis and trying to disprove it, and understanding that you have to have evidence before you jump to conclusions and corroborating evidence. So, we talked a lot about skepticism. So, it's important to be skeptical about incident response and what's potentially happening in an environment. A lot of us come into environments where everybody's panicking and we have to do a lot of crisis management. So, that becomes a second important skillset for incident responders. We, also, have to be very good at being the common voice in the room. We equated it to being a therapist or being a parent. You have to exude confidence and calm in a situation where everybody's upset. So, you have to be able to do both of those things. But in terms of investigative mindset, you have to understand that you can't jump to conclusions. Everything that you are finding has to be corroborated. And sometimes you are trying to disprove something instead of prove it. Everybody else is certain that the crisis is being caused by a particular state or it's cyber caused at all. And you're coming in saying, "Well, let's get some evidence. And what if it isn't? Let's try to disprove that actually being what happened." That's how you do good science and good investigation. So, you have to have both of those skillsets, which is a challenging combination sometimes.

Dave Bittner: Is diplomacy a part of it, too, your interactions with the folks? You're the outsider coming in, right?

Lesley Carhart: I sometimes call my job "marriage counseling," in fact.

Dave Bittner: I love that.

Lesley Carhart: And that's especially applicable to industrial incident response.

Dave Bittner: Yeah.

Lesley Carhart: Which is a different beast in a lot of different ways. But in industrial incident response, we have even more personality management. We often have the engineers and the operators who do the important process work, they are the bread and butter of their organization. And then you have a cybersecurity team as well who's doing important work to protect that process space. But they speak a different language and they step on each other's toes. Sometimes there has been a decade or two decades of hostile relationship between those groups, between IT people not understanding the process and not understanding that they can't just bring systems down to patch them. That's problematic for safety reasons, for life and safety reasons. And those miscommunications over time have built this animosity where sometimes our team just has to come in and sit at a table for them to have a conversation. We just have to sit there. We don't just say a lot. You know, we come with some donuts and we sit at the middle of the table, and then they'll finally talk to one another, which they haven't done in years. But, yeah, incident response is a lot of that. There's a lot of personality management. You have a lot of people in authority who are panicking and trying to - they might perceive that they've done something wrong, even though incidents can have - happen to absolutely any organization. But there's a lot of blame and passing blame during incidents oftentimes. And there's a lot of people trying to protect themselves and their careers. Everybody's stressed out. It's a big crisis. It's the worst day for an organization. So, a lot of what we do is try to calm people down and, again, be that authoritative calm voice in the room.

Dave Bittner: As you walk around the show floor here at the RSA Conference, what are you seeing in terms of trends in your specific industry, industrial control systems? Or are there any patterns you're seeing among the providers there?

Lesley Carhart: The interesting thing to me at RSA every year is seeing what the flavor of the year is. Everybody's kind of coming in with similar products every year. You know, they're still selling their services or their products and they do important things. But they frame their advertising and their marketing materials around whatever people are kind of worried about the collective subconscious for the year. And that's fascinating to me because it's not just the collective subconscious of technical practitioners like me, but also like executives, what are they worried about. You'll see that reflected across the floor at RSA in terms of marketing and branding and how people are selling the same things that they sell every year, but this year they're concerned about specific things. So, on the floor this year, you see like a lot of ChatGPT, you see a lot of SBOM. SBOM is very relevant to industrial cybersecurity. So, I'm really happy to see that discussed. Of course, the foundations and fundamentals are still very, very challenging for industrial operators. And is there more attention being brought to that at RSA? Not necessarily because the fundamentals, like asset management and basic security monitoring, they are just cool and fun to flashy ad advertised for at RSA definitely. But they're still very, very important. But all the things that people are concerned about every year tend to be real issues that matter for cybersecurity. Like, last year, we saw a lot of discussion of Zero Trust. Everybody was - everybody's booth, no matter what they were selling, they all talked about Zero Trust. The year before that it was MITRE ATT&CK. And those are all important things that are important elements of cybersecurity. It's just interesting to me seeing what people are concerned about and what they're talking about at RSA every year.

Dave Bittner: Yeah, absolutely. Well, Lesley Carhart is Director of Incident Response at Dragos. Thank you so much for joining us.

Lesley Carhart: An absolute pleasure. Thank you so much.

Dave Bittner: In this week's "Learning Lab," the conversation continues between Dragos' Mark Urban and Kimberly Graham. They're talking convergence.

Mark Urban: I am Mark Urban with another episode of the "Learning Lab" on "Control Loop." And, today, we're going to continue the discussion on convergence between OT and IT. When we had Rob Lee a couple episodes ago talk a little bit about convergence, he brought it into the SOC processes. And we'll revisit that, click down on that a little bit. But then also get an understanding about how OT can integrate with other infrastructure pieces that are traditionally fine on and - on the IT side. And, for that, I'm joined by Kimberly Graham, Dragos' Head of Product. Kim, welcome.

Kimberly Graham: Thank you.

Mark Urban: Let's look more maybe into a little proactive about how are you creating kind of the environment and the architecture to do better preventative, how - what's the interplay of OT and IT and, you know, how does it intersect, for example, with firewalls?

Kimberly Graham: Firewalls fulfill a very specific role. And I won't say it's terribly dissimilar between IT and OT, but the way in which you implement them winds up being different. So, it's all about network segmentation. The OT world has a very well-defined model for that. I'm assuming many of the listeners know the Purdue model. But that implementation is different depending on the industries. You know, some industries are very much fully behind the Purdue model, some industries you'll see a lot of flat networks and so on. But getting that network segmentation within the OT environment and in the segmentation that really separates things from the IT environment is really key to, you know, being able to make sure that you're defending from as many attacks that would hop from IT to OT. And then, in addition to the segmentation, you have to monitor it because firewalls are there to facilitate communications when they're intentional. But things can be left behind, rules can be left in place, rules can be misconfigured. So, having monitoring that sees that communication path between OT and IT is very, very critical.

Mark Urban: Gotcha. So, like, the Purdue model levels 1, 2, 3 in the OT domain, level 4 is typically the IT domain. And, you know, the little sandwich meat in between layer 3 and 4, sometimes referred to as 3.5, where you would typically see a firewall to kind of segment that and, you know, provide a hard line of policy demarcation between those two environments. And if a firewall serves both OT and IT, how does the firewall get the context for OT? You know, is there - is that, you know, what some of these tools can provide?

Kimberly Graham: Yeah. So, one of the things that monitoring tools can often provide is a view into what is communicating. If someone's in the state where maybe the firewalls are not as restrictive as they should be and you need to get a little more context, then you'll need a monitoring tool to help you understand how our existing communications path's working, are there any areas where it makes sense for us to consolidate. It's very easy to think of things as starting from a green field, but that's usually not the case. Usually, things are set up and running. And now it's we need to make sure that we're locking down firewalls in it in a good way. And you can't do that without operational impact unless you're actually monitoring and watching that traffic, and making sure this is traffic we want to see, you know, using baselining to say, "This is a baseline for my communications, let me see what deviates, let me watch that for a little while and see if this is something that we can start implementing and restricting." And then, depending on your firewall tooling, you may have integrations with these monitoring platforms where you can do asset enrichment or taking addresses that behave a certain way and feeding that into the firewall from a monitoring platform.

Mark Urban: Gotcha. Feeding that into the firewall isn't - you know, because firewalls are primarily a policy construct. And one of the discussions that we have, Kim and I have had this discussion also, is about typically in OT systems when you get into the actual operational environment, you're talk - you know, Kim mentioned earlier that these are typically closed systems with certain expectations about how communications take place. Often, they're running at the edge of their capacity and can be very brittle environments. So, typically, we see a lot of passive monitoring in those environments, you know, within that OT 1, 2, 3 level. If the firewall is at, you know, between the 3 and the 4, and it is a policy place, Kim, is that where we can see - you know, is there a nexus of where you can do policy enforcement, you know, for O - to protect OT, even if you're not doing any kind of active policy within the actual OT guts? If that makes sense.

Kimberly Graham: Yeah. So, even if there's not active enforcement, having a monitoring tool that allows you to create zones and create groups of devices that communicate into that Purdue structure is important to have to understand that devices are communicating in the ways that that you expect them to. So, even if you don't have points of enforcement, you don't have, you know, maybe micro segmentation or something deeper into that OT environment, you can at least start seeing if there's communication paths that just don't make sense. In these highly segmented environments where the OT environment is very well segmented from IT, there's very good supportive monitoring of ingress and egress, there is, you know, a secure mode access tool to make sure that there's monitored and controlled access into that environment, then a lot of times it is okay to, you know, rely fairly heavily on monitoring to make sure that communications are happening in the way they can if introducing additional controls is going to introduce additional points of failure or risk that environment. So, just like everything in OT, it's usually a risk decision of how much more enforcement do I want to add versus how much more monitoring do I want to add. And as long as you can do that monitoring in a passive way, then that monitoring is usually the safest route in terms of operational impact.

Mark Urban: Gotcha. And I think what I was getting at - so, that - that's a - that's, you know, a good lesson, passive bonding within the guts and, you know, that can interact with the firewall to - you know, if there's policy that can be helpful from, you know, terminating an external connection, et cetera, that's where we can look to a firewall to do policy enforcement that's not doing any active things down in the OT network to -

Kimberly Graham: Absolutely.

Mark Urban: To protect it.

Kimberly Graham: Yeah. And it's not that, you know, I think that the OT space is ready for a full soar for doing full automation and orchestration of certain types of responses, like gating. But when you're talking about things like restricting access between IT and OT and working in a more restricted mode, I think that type of world does make sense for using certain types of threat behaviors that are detected to maybe trigger that OT environment to work in more of a restricted scenario rather than an open scenario. And you can design those scenarios in ways that don't impact the operations.

Mark Urban: Right. That's - you know, if we - if firewalls - that - that's a good description. Maybe if we move to what about endpoint technologies, do we see a lot of endpoint technologies in the OT space, I assume because, you know, those devices have to run on something, they are endpoints. But maybe describe a bit the interaction of kind of endpoint security within those - you know, within an OT environment and how that fits in.

Kimberly Carhart: Sure. So, you are going to have somewhat traditional types of endpoints in different parts of the OT environment. You're going to have engineering workstations that are almost always Windows workstations. And those are like traditional endpoints. Now, they're in a different environment. They're going to be treated differently. And, often, they are under a support contract with a very specific OT vendor. In those cases, you're limited in what kinds of endpoint technologies you can use. But we do see some, I'd say, growing different types of endpoints in that space. So, it used to be more on just the antivirus side and now we're seeing more EDR, like CrowdStrike and others, on those more traditional IET style endpoints, even though they're really OT assets.

Mark Urban: Yeah. But if - you know, one can imagine that, you know, their traditional IT solutions that you'll find down there, they probably miss a lot of, again, the OT context that might be helpful. Is there - can you leverage, like, an EDR to - you know, on the OT side, can you get information from there that's helpful to an OT kind of monitoring?

Kimberly Graham: So, you're right, the EDR technologies that are provided by the IT space tend to be very IT focused. And that's fine. These are endpoints, they do exist in ways that IT assets exist. But to your point, it's not just the IT side. These are engineering workstations that interact with PLCs and other devices. So, they're very different than a traditional workstation that would be on the IT side. So, you can have integrations that allow you to share the best of both worlds. So, looking at the asset profile that maybe the EDR endpoint software can collect, using that to enrich the data in the passive monitoring system. And then also correlating events. So, taking events that are seen in one system, pushing it to the other system, doing consolidation and really understanding the full end-to-end story of a specific type of attack. Because if they are impacting that specific engineering workstation, then what's happening with that EDR is also interesting when compared to what's happening on the network. And since those types of endpoints are - tend to be these more like traditional IDS, that's - you see things like encryption, you see things like lockdown services. So, just doing a sole passive monitoring has the potential to miss certain things. All the communications will be seen, but what's inside of it, sometimes you do need to be able to integrate with those types of agents to be able to see that.

Mark Urban: Those EDR systems, because they're sitting on the endpoint, can provide a lot of good device-specific information into the OT monitoring tool that is based on passive. But, similarly, the passive monitoring tool can detect things across a network in an OT context that that EDR would probably not have the capability to detect. But it could also, you know, look at specific behaviors to help, you know, correlate other things happening to simplify the root cause analysis. So, it seems like a good Ying and a - Ying and a Yang to - you know, of OT and IT to merge that passive monitoring and that endpoint security. Kim, thank you so much for your insights and everybody at "Control Loop." Thanks again for listening.

Dave Bittner: And that's "Control Loop" brought to you by the CyberWire and powered by Dragos. For links to all of today's stories, check out our show notes at the cyberwire.com. Sound design for this show is done by Elliot Peltzman, with mixing by Tré Hester. Our Senior Producer is Jennifer Eiben. Our Dragos producers are Joanne Rush and Mark Urban. Our Executive Editor is Peter Kilpe. And I am Dave Bittner. Thanks for listening. We'll see you back here next time