Control Loop: The OT Cybersecurity Podcast 6.1.22
Ep 1 | 6.1.22

Welcome to Control Loop: Giving back to the OT community.


Dave Bittner: Welcome to the first episode of Control Loop, our show about industrial cybersecurity news, strategies, and learning. We’ve partnered with the OT cybersecurity experts at Dragos to bring you this show every two weeks. Enjoy.

Dave Bittner: It’s June 1, 2022 and you are listening to Control Loop. I’m Dave Bittner. In today’s OT cybersecurity briefing:

Dave Bittner: Continuing expectations of escalation in cyberspace during Russia’s hybrid war against Ukraine. Russian threat actors appear to be preparing attacks against industrial control systems. Proof-of-concept exploits for Bluetooth Low Energy systems. CISA and international partners recommend best practices (and they’re relevant to OT as well as IT).  Hacktivists claim they’ve counted coup against Russian ground surveillance robots. DDoS as potential battlespace preparation. A new wiper loader reported. Turla threat actor found conducting reconnaissance in Estonian and Austrian government networks.

Dave Bittner: I speak with Rob Lee about his plans to give back to the OT community, the big idea behind this show, and his candid thoughts about the Pipedream malware and its creators.

Dave Bittner: And later in the Learning Lab, Mark Urban and Jackson Evans-Davies teach us about the fundamentals of network security for operational technology.

Dave Bittner: Microsoft President Brad Smith, speaking in London at the Microsoft Envision conference, renewed calls for laws of conflict in cyberspace, Infosecurity Magazine reports. The rules he envisions are essentially transpositions of traditional jus in bello considerations: proportionality, discrimination, and the avoidance of perfidy. They're none the less sound for being familiar. Smith sees the hybrid war in Ukraine as having lent new urgency to the development of international norms.

Dave Bittner: The cyber phases of Russia's hybrid war have shown some correlation with kinetic operations, but less than many had expected. PCMag describes the ways in which cyber operations appear to have been conducted without close coordination with conventional forces. Some of the cyber phases of Russia’s war have shown an interest in targeting industrial systems, even if the widespread and devastating attacks against infrastructure many predicted have not yet materialized.

Dave Bittner: For example, there are strong indications that a threat actor has targeted industrial systems in Ukraine. And circumstantial evidence points to Russia. The US Government hasn't made that attribution, but several security companies have.

Dave Bittner: While Dragos doesn’t assign attribution to threat actor groups, it did report the discovery of an ICS-focused attack toolkit designed with the energy sector and particularly liquified natural gas facilities as their targets. Dragos named the attack kit PIPEDREAM and the threat actor group CHERNOVITE. The unusually large number of industrial control system advisories the US Cybersecurity and Infrastructure Security Agency (CISA) has released recently seems a partial response to this recently discovered threat.

Dave Bittner: In mid-April CISA warned that "certain advanced persistent threat (APT) actors have exhibited the capability to gain full system access to multiple industrial control system (ICS)/supervisory control and data acquisition (SCADA) devices using custom-made tools." The vulnerable systems include at least Schneider Electric programmable logic controllers (PLCs), OMRON Sysmac NEX PLCs, and Open Platform Communications Unified Architecture (OPC UA) servers. The advisory recommends familiar best practices for protecting ICS/SCADA systems, and explains the threat actor's tools as follows:

Dave Bittner: "The APT actors have developed custom-made tools for targeting ICS/SCADA devices. The tools enable them to scan for, compromise, and control affected devices once they have established initial access to the operational technology (OT) network. Additionally, the actors can compromise Windows-based engineering workstations, which may be present in information technology (IT) or OT environments, using an exploit that compromises an ASRock motherboard driver with known vulnerabilities. By compromising and maintaining full system access to ICS/SCADA devices, APT actors could elevate privileges, move laterally within an OT environment, and disrupt critical devices or functions."

Dave Bittner: The immediate actions CISA recommends are to implement multifactor authentication, change system passwords (especially any default passwords), and use "a properly installed continuous OT monitoring solution to log and alert on malicious indicators and behaviors."

Dave Bittner: The Washington Post reports expert consensus that the energy sector, especially liquefied natural gas facilities, are the tools' most likely targets.

Dave Bittner: While CISA's advisory called out specific products and merely suggested that others might be vulnerable, Dragos is explicit in its assessment of CHERNOVITE’s PIPEDREAM that other systems are at risk: "the tooling may be used to target and attack controllers from hundreds of additional vendors. PIPEDREAM can target a variety of PLCs in multiple verticals due to its versatility." That versatility has been observed elsewhere. Wired quotes sources at Dragos to the effect that PIPEDREAM is “like a Swiss Army knife with a huge number of pieces to it.” It's equally capable of collection, compromise, disruption, and destruction of industrial systems. Two of the points Dragos makes illustrate the versatility: "CHERNOVITE can manipulate the speed and torque of Omron servo motors used in many industrial applications and whose manipulation could cause disruption or destruction of industrial processes leading to potential loss-of-life scenarios. PIPEDREAM’s Windows related components facilitate host reconnaissance, command and control, lateral tool transfer, and the deployment of unsigned rootkits." The warnings about this threat to control systems are forward-looking, as the tools don't appear to have been used, yet.

Dave Bittner: On May 7, 2021, DarkSide ransomware operators gained access to a VPN at Colonial Pipeline, a major energy supplier to the eastern United States. The gang succeeded in encrypting data and disrupting Colonial’s ability to track the flow of product through its pipelines. Pipeline safety wasn’t compromised, and Colonial was able to restore its systems within a matter of days. Looking back a year later, Utility Dive reviews some of the lessons learned from the incident.

Dave Bittner: The attack showed the effect that an attack on business systems can have on industrial processes, even when there’s no direct interference with the OT systems proper. And, while modernization of OT systems has produced efficiencies and economies, it’s also opened up attack surfaces that were hitherto unknown in older, legacy, air-gapped analogue control systems. The response to the incident in both the public and private sectors has been on the whole a positive one, marked by greater cooperation, by more sharing of threat intelligence, and of a heightened awareness of the importance of public-private cooperation.

Dave Bittner: Cooperation in cybersecurity was also a theme at this year’s meetings of the World Economic Forum in Davos. CISA Director Jen Easterly expressed high expectations for Davos. “Cyber is the ultimate borderless space—a global challenge that mandates global solutions,” she tweeted. “Looking forward to this week at #WEF22 @Davos to discuss the importance of global operational collaboration among government & industry partners for a more secure & resilient cyberspace.”

Dave Bittner: Her expectations were not disappointed. A prominent feature of this year’s proceedings was the circulation and widespread adoption of the Cyber Resilience Pledge, a resolution that involves not only a commitment to greater resilience, but also  a commitment to placing security outside the bounds of competition. The pledge is expected to have a positive effect on organizations, from the operator levels to the board room.

Dave Bittner: The World Economic Forum explained the importance of the pledge, which committed major representatives of the energy sector to the pursuit of greater resilience. “For the first time, 18 global organizations from the oil and gas ecosystem are championing a unified approach to mitigating growing cyber risks and pledging to promote cyber resilience. The global cost of cybercrime is expected to reach $10.5 trillion a year by 2025; the threat of infrastructure breakdown due to a cyberattack is the top personal concern for cyber leaders.”

Dave Bittner: The pledge begins with oil and gas, but the World Economic Forum hopes to see comparable resolutions, and attendant action, from other economic sectors. The Forum’s Global Cybersecurity Outlook 2022 provides a template those other sectors can use as they advance their own resilience.

Dave Bittner: NCC Group researchers have demonstrated that Bluetooth Low Energy (BLE) systems are vulnerable to link layer relay attack. The news has been generally reported with headlines that point out that crooks could now open and start your Telsa without so much as a by-your-leave, but the problem is more widespread than that. BLE is, NCC Group explains, "the standard protocol used for sharing data between devices that has been adopted by companies for proximity authentication to unlock millions of vehicles, residential smart locks, commercial building access control systems, smartphones, smart watches, laptops and more." It's not the kind of problem that can be resolved with a patch. Rather, NCC Group argues, it's the kind of issue that arises when technologies are extended beyond their intended purposes, and BLE, they say, was never designed for use in industrial infrastructure. The researchers offer three recommendations, two for manufacturers, one for users:

Dave Bittner: They say"Manufacturers can reduce risk by disabling proximity key functionality when the user’s phone or key fob has been stationary for a while (based on the accelerometer).

Dave Bittner: They say "System makers should give customers the option of providing a second factor for authentication, or user presence attestation (e.g., tap an unlock button in an app on the phone).

Dave Bittner: And, ”Users of affected products should disable passive unlock functionality that does not require explicit user approval, or disable Bluetooth on mobile devices when it’s not needed.”

Dave Bittner: The US Cybersecurity and Infrastructure Security Agency (CISA) and its partners in Canada, the Netherlands, New Zealand, and the United Kingdom recently issued Alert (AA22-137A) "Weak Security Controls and Practices Routinely Exploited for Initial Access." The Alert describes "common weak security controls, poor configurations, and poor security practices" that are used for initial access, and it recommends particular attention to seven best practices. These are relevant to industrial control systems as much as they are to IT systems generally considered.

Dave Bittner: They include, ”Control access.

Dave Bittner: "Harden Credentials.

Dave Bittner: "Establish centralized log management.

Dave Bittner: "Use antivirus solutions.

Dave Bittner: "Employ detection tools.

Dave Bittner: "Operate services exposed on internet-accessible hosts with secure configurations.

Dave Bittner: and "Keep software updated."

Dave Bittner: The rating service Fitch notes the significance of OT security to utilities’ credit and ESG (environmental, social, and governance) ratings. It sees an attack on OT systems as likelier to be more consequential than a comparable attack on IT systems. 

Dave Bittner: Attacks on the power, water, and sewer sectors have risen, and  so has the interconnection between their OT and IT systems. Utilities can no longer rely upon the traditional segregation of OT from outside access. The loss of legacy air-gaps and other separations has proceeded apace as obsolescent systems are replaced with newer, networked controls, and the rating agencies are putting utilities on notice that their adoption of best practices will be a matter of interest.

Dave Bittner: The Daily Dot reports that a hacktivist group, "CaucasNet," says it successfully compromised Tral Patrol 4.0 unmanned ground video surveillance systems. Hashtagging #OpRussia and #GloryUkraine, CaucasNet's Twitter feed crowed, "We hacked the patrol robots of the Russian company «SMP Robotics». Now we control the Robotics robots all over the world, we broadcasted the anthem of Ukraine and the Georgian song «300» on all the robots on May 9th." Tral Patrol robots have been sold in many countries, but CaucasNet claimed in particular that they'd hacked the systems at Moscow's Sheremetyevo International Airport. The airport did not confirm any incident to the Daily Dot, saying only, “Sheremetyevo International Airport does not confirm the fact of hacker hacking of the security system." Like most hacktivism, this amounts to a nuisance. And like most hacktivist claims, this one should be received with open-minded skepticism.

Dave Bittner: An Iranian group has claimed responsibility for a distributed denial-of-service (DDoS) attack that interfered with the Port of London Authority's website. The Authority acknowledged the incident but said that operational systems were unaffected. The group that said it was behind the attack, the ALtahrea Team, is a nominally hacktivist group, HackRead says, that operates under the direction of the Iranian government. While the incident affected, apparently, only a public-facing website, and not port systems involved with scheduling, material handling, or intermodal transfer, such incidents serve as indicators of adversary interest, and can also be used as misdirection for more disruptive or destructive attacks. 

Dave Bittner: The GRU's Sandworm group, ESET reports, has deployed a new version of its ArguePatch loader. ArguePatch had seen previous use in both Industroyer and CaddyWiper attacks against Ukrainian targets. "The new variant of ArguePatch – named so by the Computer Emergency Response Team of Ukraine (CERT-UA) and detected by ESET products as Win32/Agent.AEGY – now includes a feature to execute the next stage of an attack at a specified time. This bypasses the need for setting up a scheduled task in Windows and is likely intended to help the attackers stay under the radar."

Dave Bittner: BleepingComputer reports that the Russian threat actor Turla, also known as Snake or Venomous Bear, and associated with the FSB, has staged typosquatting domains for use against Austrian and Estonian targets. The activity so far represents a cyber reconnaissance phase of battlespace preparation. It is, as the Sekoia researchers who discovered it say, a phishing campaign:

Dave Bittner: "SEKOIA.IO Threat & Detection Research (TDR) Team have expanded the search on Russian-linked TURLA’s infrastructures from a Google’s TAG blog post. It exposes a reconnaissance and espionage campaign from the Turla intrusion set against the Baltic Defense College, the Austrian Economic Chamber which has a role in government decision-making such as economic sanctions and NATO’s eLearning platform JDAL (Joint Advanced Distributed Learning) pointing to? Russian Intelligence interest for defense sector in Eastern Europe and for topics related to the economic sanctions against the Russian Federation."

Dave Bittner: And, finally, this week OT cybersecurity practitioners will converge on Orlando for the SANS ICS Summit. The conference is timely, coming as it does during a period of both heightened risk and heightened awareness of OT threats. The summit will feature training, expert presentations, and opportunities for professional networking. Stay tuned for reporting from the conference…

Dave Bittner: And it is my pleasure to welcome to the show Robert M. Lee. He is the CEO at Dragos. Rob, I am excited to say that we are heading off on a collaborative project here, the "Control Loop" podcast sponsored by Dragos. And you all are, of course, heading up large parts of this effort. Let's start with some basics here. Why this? Why now? 

Robert M Lee: Yeah. So good to collaborate with you. 

Dave Bittner: (Laughter). 

Robert M Lee: I feel like we've been in orbit with each other for a while, so it's good to put a ring on it, as Beyonce would say. 

Dave Bittner: Yeah. 

Robert M Lee: And so when I look at why now, the reality is OT security has become such a main topic now. It is truly a global, up from executives on down to practitioners, discussion. It's not just this little community that we've been - you know, a decade ago we could all sit around the fire literally at a conference and know everybody around us. Now it's much bigger, which is awesome. But with that comes a lot of information overload, and there is a lot of good guidance getting out there, and there is a lot of bad guidance. And there's also just too much information sometimes for anybody to reasonably consume when you're busy day to day. So why now? Because there's that plethora of information we can synthesize down. Here is the things that you need to be aware of. What we're hoping to accomplish with it is exactly that. I'd like to make the podcast kind of two things, and that's what you and I have talked about for a while. The first thing is kind of the news capturing of all the different stuff out there, of all the new papers, of all the new research, of all the new news bites, what's your 15-minute-or-so digest of this, and just make this accessible to people. I mean, again, we're all overly busy. Just to be able to have audio for a commute or even just preparing around the house for the morning, to be able to synthesize all that information, that's a good service to provide to people. And so that's part of it. 

Robert M Lee: The second part is we are welcoming in a significant increase of percentage of professionals into the OT community versus what's there today. In other words, you onboard 500 new people into InfoSec, doesn't put a dent in the size of InfoSec. You onboard 500 new people into OT security, that's a significant contribution to the percentage of the current state of the community. And so we need to have a forum of sorts to kind of, like, onboard them and make sure that they are getting some basic concepts and understanding. So the second half of the "Control Loop" podcast, if you will, is meant to just be a very educational, hey, here's how a control loop works. Hey, here's what a gas turbine is and where you might find them and what they do. Hey, here's why OT is different than IT. So just have the - kind of these educational things. And I think as we've talked, the idea is to launch each episode in its full but to take that second half of the episode and create a library of content for people that can come back and just uplevel their knowledge of ICS security. 

Dave Bittner: Who's the target audience here? I mean, obviously, we want folks within OT security to listen. But it strikes me, like, you know, there's a lot here for folks who are outside of that specific community as well. 

Robert M Lee: Yeah, I think the first half will be kind of an everybody thing. And I hate to say it that way, but it really is. There's nobody out there that's not interested in what's happening in our infrastructure security and kind of being up to date with the news. And if you're trying to keep up to date with everything, you can't, but a 15- to 30-minute digest of, here's the stuff you need to know, literally, you'll have not only CSOs and executives and practitioners and all that, but you're going to be bankers and financial analysts and market analysts and everybody else trying to keep up to date. So I think it's going to be a lot wider than people realize, in that first part. That second part will be more practitioner focused. That will be where you've got the - maybe the CSO who's trying to get more familiar with what programs are about to roll out, but definitely IT security professionals trying to onboard into operations. Like, I think that'll be the core segment, our core audience for that portion of the show. 

Dave Bittner: All right. Well, I want to touch base with you today about everything going on with Pipedream, this ICS-focused malware that you and your colleagues have had a hand in the discovery of. But I think there's a lot to the story here. Where's a good place to start? 

Robert M Lee: Yeah, I would just give a background for folks to say that, when you look at industrial control system-focused attacks, most of what we worry about on a day-to-day basis is the abuse of native functionality. It's not about some malware. It's not about some vulnerability. Actually, vulnerabilities tend to be a very system-based view of the world in the world of industrial systems, of systems and physics. So it's less about what can you do to one system; it's much more about, do you know how to operate a circuit breaker? Do you know how to operate a gas turbine? Do you know how to operate these different systems of systems that we have? And if so, you can abuse that functionality to do disruptive effects. But every now and then you actually get ICS-focused malware. And they largely, so far, have come in kind of two flavors. One is access. BlackEnergy 2 is a great example of that. It had exploits for internet-facing human machine interfaces, basically being able to get access to these industrial environments. In of itself, couldn't disrupt or destroy anything, but it could help you get access. 

Robert M Lee: But then you also have the disruptive and destructive type capabilities, right? We had Stuxnet. We had CRASHOVERRIDE or Industroyer. There's Industroyer2, TRISIS. These ones are deployed to do something disruptive or destructive. And across all of those cases and across all the time that we have, there's only been six publicly known ICS malware toolsets, and most of them are really victim-specific. You're really not going to use it somewhere else. The playbook that they've now shown, the tradecraft that they've shown can be picked up by other people. But you're not just going to dropship it into another environment. TRISIS, as an example, worked against that petrochemical environment, with that safety system. The things they exposed - anybody can now copy their playbook. But you're not going to see TRISIS in its current form deployed somewhere else. 

Robert M Lee: And that brings us to Pipedream. So Pipedream is, in my opinion - I hate this whole like, who's the best? You know, what's the most sophisticated malware? I don't want to get into that measuring contest crap. It doesn't matter. But what we can candidly say is Pipedream is the most flexible of the ICS capabilities we've seen. So anything new - right? - the seventh ICS malware framework is going to be big news anyways. But the fact that it can go against such a wide variety of industries and equipment makes it particularly dangerous. And what's probably most interesting to people around the world is we were able to get this information out to people and analyze it before the adversary employed it on its target. It's not saying they haven't deployed it anywhere in the world. It's not like it's not out there somewhere, but it wasn't employed against their actual targets. And I'll pause there for a second. 

Robert M Lee: But in our view, in our assessment, this was a capability designed to be disruptive, if not destructive, against a set of initial targets and then capabilities beyond that. What I mean by that is this looks like they were going to deploy it against U.S.-based energy assets, specifically in the liquid natural gas space, both electric and gas community. I mean, I honestly think that they were going to use this. And when you talk about attacks on U.S. infrastructure in a reliable way, I mean, that's something - there's many people out there that were like, oh, we're not going to get attacked. We're not at war. Blah, blah, blah. 

Robert M Lee: I was like, yeah, the adversary gets a vote in that, you know? And this was very, very bold and brazen. So we're fortunate we found it beforehand. But there's no fix to it. It's not like there's a vulnerability they're exploiting. It's not like there's something that you can just go patch and fix. They're doing all the things we've been warning about for years - using Modbus/TCP, a very common ICS protocol, using OPC, a very common ICS protocol, exploiting CODESYS functionality, which is software in just hundreds of different controllers out there. So it's one of those capabilities that, if I was building an ICS security program from scratch and you just modeled out this scenario and protected yourself against it, from protection, detection and response mechanisms, you would have a world-class program. Like, this is a very capable framework. 

Dave Bittner: I think there's been a lot of attention to the fact that your team and some other teams, folks at Mandiant as well as your team at Dragos, were proactive on this, were able to, as you mentioned, you know, have the detection before it was deployed. You know, you went so far as to take the stage and kind of give these threat actors, you know, a bit of the riot act about their capabilities. And you draw some attention to that or - I mean, there was attention on you because of that. 

Robert M Lee: Yeah. 

Dave Bittner: Why take that approach? Is that putting a target on your own back? 

Robert M Lee: Probably. And so, look, I don't think anybody's above critique or reproach. And so I'm happy to have anybody try to critique me on any of my statements and actions. Why - I think you're alluding to my response on Twitter, to my keynote. What I kind of push back on is there were people that weren't in my talk that were then tweeting at me about their opinions of what they perceived to be my stance. And so first, I was saying, hey, guys, watch the video, or watch the talk before you come at me. 

Dave Bittner: Right. 

Robert M Lee: And number two - you know, and I don't mean this in any arrogant way. I don't mean this to be braggadocious. I don't mean this to be a jerk. But I have been on the offense for this country. I have been on defense. I built the ICS threat discovery mission for the government. I run the largest ICS security company in the world right now over at Dragos. I'm not saying I'm right, but I think I have experience enough to make the statements that I make. And for people to be like, Rob, it's bad that you're poking the adversary. Guys, I've been there, done that. You may not agree with me, but I'm precise with my words, and I know what I'm saying. And so why did I say that, right? 

Robert M Lee: At the end of the talk, I put down the adversary. Why? To me, this community - and I love them to death, and there's plenty of reasons to do it, don't get me wrong - this community builds up adversaries to almost hero worship, to a fact - to a side, for me, that feels disgusting. We're so happy to talk about, oh, this is the most sophisticated group, and, oh, these people were amazing. Did you look at this cool hack that they pulled off? Or let's memorialize them with statues at RSA for the various threat groups that they represent, and all this crap. And it's honestly kind of disgusting to me, personally, because having been on that side of the world. And having been in the intel community, I know for a fact many of the developers and operators of these campaigns just absolutely revel in that. 

Robert M Lee: It's a glorification. It's a, hey, did you see the latest report they were writing about our team? Look how great and wonderful we are, etc., etc., etc. So my intent was to kind of return a little bit of normalcy and say, you know what? As a member of the industrial community, out to the adversaries here, I just wanted to let you know, we don't think you're clever. We don't think you're cool. You're going after civilian targets and civilian people, and you should feel bad. You should be fired for your incompetent approach to this. And I think they ought to be reminded every now and then that they're not as important or as cool as people make them out to be. They are jerks trying to hurt people. And in any world, in any country, in any reality, I hope all of us can agree that civilians should be off limits. 

Dave Bittner: All right, Robert M. Lee, thanks for joining us.

Dave Bittner: In today's Learning Lab, we have Mark Urban and Jackson Evans-Davies. Mark, welcome. Hope you're doing well. How about you introduce yourself to our audience? 

Mark Urban: Hi, Dave. Thanks. Yes, I'm Mark Urban, vice president of product and industry market strategy here at Dragos - 20 years in cyber across a number of different areas and have worked a lot with CISOs and architects to look at the architecture of their environments to make it more secure. 

Jackson Evans-Davies: Hi, Dave. Yeah, great. Thanks for having me as well. My name is Jackson Evans-Davies. I'm a principal penetration tester with Dragos. And today we're going to be talking about securing an industrial environment from a high-level perspective. 

Mark Urban: Now, Jackson, we've talked a lot about this subject. And, you know, I think, when we look at connectivity into the industrial side of an organization, that can be very problematic. What are the things that drive, you know, people wanting to connect into the industrial side of an environment? 

Jackson Evans-Davies: Yeah, Mark. I think that's great - a great question. A couple of things we see are third-party vendors needing access into the environment to fine-tune their equipment. And the other would be the asset owners and operators themselves. They would need access in to, you know, monitor and, you know, fine-tune it as well. And we've seen an increase of that since the start of COVID about two years ago. With everyone working from home, organizations created remote access solutions. Some were secure. Some were not. And over the last couple of years, they've slowly started to secure those environments. And it's something that, you know, we see continue going forward. And organizations kind of need to get on board with, you know, defending that remote architecture. 

Mark Urban: So by remote access, we're talking about VPNs and other secure - you know, quote-unquote, "secure access technologies" that - you know, whether you're - it's within the organization coming from home or whether it's a third-party supply chain vendor coming into the environment that's VPN, secure access - what are the right ways to architect that remote access environment at a high level? 

Jackson Evans-Davies: I think what we need to look for when we're talking remote access or VPN connections, like you mentioned, into the industrial environment is really break it down from what the Purdue model shows us. The Purdue model shows us Level 4 being corporate, Level 3.5 being the DMZ, Level 3, 2 and 1 being the industrial or OT process underneath it. So the way we'd like organizations to create remote access into the OT environment, whether it's themselves or third parties, is come in through corporate Level 4 VPN connectivity or RDP connections and then down into the DMZ environment and then down again into the OT environment - all of that using multifactor authentication at two locations, between IT and OT, and then OT and down below into the ICS environment - and also using different sets of credentials at each level as well. It adds a wealth of sophistication to an environment. 

Mark Urban: So you're saying that you want a DMZ sandwich, so to speak. You've got the - kind of the IT environment, where people kind of come in. Then over in another area, you have the OT, the industrial kind of firewall access. And in between those - what goes in between those two firewall DMZs? 

Jackson Evans-Davies: Yeah. So that's what we call the DMZ level itself. And in the DMZ environment, we often see, you know, a series of connections coming in. Often a jump host is configured in there. There's domain controllers - 'cause we want a separate domain in the DMZ as well, not a DMZ or a domain span of the corporate or a domain span of the OT environment. We want an isolated domain in the DMZ. And then we also see - you know, on the industrial side of the DMZ, we see Py connections coming up. We see historians in the DMZ as well, to be able to push that data up to corporate environment. So the DMZ is kind of a combination of both assets, corporate and OT, and that allows us to segment those communication paths, you know, as - we could consider almost a waterfall - corporate to DMZ and then DMZ to OT and then vice versa going back up. 

Mark Urban: So coming in VPN into the OT - or, I'm sorry - into the IT DMZ, and then you're looking at - there are ways to think through domain strategies. There are specific hosts or jump hosts to segment. And then there are access policies that you can have, including multifactor authentication. But you're also - talked about - there's some advanced things that you can do with, you know, just segmenting networks and IP spaces by vendor. Can you talk a little bit about, at a high level, what that looks like? 

Jackson Evans-Davies: Absolutely. Yeah. I think that was one thing we could do. You mentioned the jump host being a pivot point in and out of the environment. It's how everyone would get, you know, from IT to OT, is through that DMZ jump host. You know, that includes asset owners and third parties. You know, and taking the architecture a step further, we could have two separate jump hosts - one for internal employees and then one for third-party vendor employees. That way, credentials are not being joined together within one asset - 'cause an adversary will look for hashes or credentials on a specific machine if they get local administrator. And if both vendor and internal share one jump host, they would get access to all the credentials on that one machine. So isolating those two jump hosts could allow us to limit an attack to one specific area and not compromise the other. 

Mark Urban: Now, when we were setting up, when we were discussing this subject and you were talking about your day-to-day being a pen tester, what do those engagements look like? I mean, where do you go to in these environments? What's the best kind of cookie jar that you like to raid? 

Jackson Evans-Davies: Yeah. I think that really depends on the environment. You know, we start the penetration test from a Whitebox perspective. So we understand the subnets. We understand the domain. We understand the IP addresses that we - that are in scope that we can target. But we're looking for misconfigurations that we - that are often seen in ICS environments. And that was kind of shown through the - you know, the Dragos Year in Review. 

Jackson Evans-Davies: Network segmentation and domain segmentation are really big ones. Those are often spanned across the environments, and we often see two-way trusts. And then one other thing we're looking for often in a penetration tester - you know, files that users leave behind - you know, whether it's on the jump host, where I mentioned people pivot in and out of the environment. Folks usually like to copy files to it, to then copy files, you know, from it. So if we can get access to a jump host that's not secured or hardened by GPO, we often go there to pull sensitive documentation, and that allows us to get a better understanding of the environment there as well. 

Mark Urban: So DMZ sandwiches, proper VPN access controls, jump hosts, good access control lists - you know, a lot of good approaches to secure connectivity into the industrial side of it. Hard to keep track of here. If you look in the show notes, there will actually be a link to a more detailed blog that even has kind of a one-picture summary of what we talked about here that you can hand off to your network architecture, you can evaluate yourself. Jackson, any kind of last thoughts on, you know, the - like, the keys to securing this environment? 

Jackson Evans-Davies: I think the biggest thing that, you know, I can touch on quick is defending the IT-OT perimeter, that firewall or that perimeter control between your corporate and the DMZ. In my opinion, that's the key to the industrial environment. If we limit data flows to and from the corporate environment to the DMZ, if we limit remote access to only those who need it and we define where they can go with unique credentials and MFA, limiting file copy between corporate and DMZ - I think those are all things that we can look at as securing that perimeter. And that's typically how an adversary will come in, is from corporate down. And that will really go a long way with defending your environment. 

Mark Urban: Jackson, that's awesome, because, you know, as we see the - you know, a lot of those persistent VPN connections into - you know, into that IT area are the starting point for most industrial intrusion. So very helpful. That's great stuff. And, Dave, we bring it back to you. 

Dave Bittner: That's Mark Urban and Jackson Evans-Davies.

Dave Bittner: And that’s Control Loop, brought to you by the CyberWire and powered by Dragos. For links to all of today’s stories, check out our Show Notes at Sound design for this show is done by Elliott Peltzman with mixing by Tre Hester. Our senior producer is Jennifer Eiben. Our Dragos producers are Joanne Rasch and Mark Urban. Our executive editor is Peter Kilpe. And I’m Dave Bittner. Thanks for listening.