Asset inventory: Part of ICS network visibility and monitoring.

Dave Bittner: It's Wednesday, May 3rd, 2023 and you're listening to "Control Loop." In today's OT cybersecurity briefing we look at how hacktivists have targeted farm irrigation systems, achieving maritime cybersecurity JCDC and pre-ransomware notification, and the case of ransomware at Fincantieri Marinette Marine. Today's guest Mike Hoffman is technical leader for global services at Dragos and a SANS instructor. Mike will be discussing IT and OT misalignment. The learning lab has Dragos' Mark Urban joined by Dragos' senior product manager Jordan Wilkerson to dig in to ICS network visibility and monitoring which is the third of the SANS institute's five ICS cybersecurity critical controls.

Dave Bittner: As part of hashtag opisrael, the anonymous affiliated cyber action directed against Israel every April and intended mostly to support Palestinian interests, hacktivists turned again to water treatment and supply facilities. An attempt to disrupt the wastewater treatment facility was blocked by the facility's security systems, but the hacktivists had more success against poorly protected farm irrigation systems. "CPO Magazine" reports the attackers had more luck with farms in the Jordan valley despite a seeming heads up from Israel's national cyber directorate. Arms in the area that did not heed the call to temporarily disable remote connections had their automated irrigation systems disabled for a time, forcing them to switch to manual irrigation. Some security researchers believe that the farms that were hit were using default passwords, making it trivial for the attackers to walk in. Again exposure to the internet and lapses in basic digital hygiene can have out sized effects on automated systems. We've seen that the U.S Transportation Security Administration has issued an emergency cybersecurity amendment for the security programs of airport and aircraft operators. Since then the Cyberspace Solarium Commission 2.0 has published a report calling for the Cybersecurity and Infrastructure Security Agency to set up a maritime equipment test bed to enhance maritime cybersecurity, FedScoop reports. The report states the program can begin by testing for cybersecurity vulnerabilities in foreign manufactured cranes used in U.S ports as mandated by the National Defense Authorization Act of the fiscal year 2023 and then expand in to broader systemically important maritime OT. CISA and the U.S Army Corps of Engineers engineer research and development center, last month released the maritime transportation system resilience assessment guide. The guide focuses on physical, cyber, geographic, and logical resilience. CISA's Dr David Mussington, executive assistant director for infrastructure security, stated that the guide is integral to the development of a unified approach to address resilience indicators for port infrastructure systems and functions that assess the key dimensions of critical infrastructure in the maritime domain. Ransomware remains a matter of concern to U.S cybersecurity officials. CISA's joint cyber defense collaborative is cultivating its pre-ransomware notification capability. JCDC stated with pre-ransomware notifications organizations can receive early warning and potentially evict threat actors before they can encrypt and hold critical data and systems for ransom. The JCDC is a public private sector information sharing organization established by CISA in 2021. JCDC associate director Clayton Romans explained in a blog post that pre-ransomware notifications are possible due to tips from the cybersecurity research community, infrastructure providers, and cyber threat intelligence companies about potential early stage ransomware activity. Romans added that, "Since the start of 2023 we've notified over 60 entities across the energy, healthcare, water, and wastewater, education, and other sectors about potential pre-ransomware intrusions, and we've confirmed that many of them identified and remediated the intrusion before encryption or exfiltration occurred." The ransomware threat to industrial operations is neither idle nor trivial nor purely theoretical. And while ransomware is most familiar as a threat to business systems, that threat can affect industrial operations as well. In this case, however, the data affected involved industrial processes themselves. On April 12th the Wisconsin ship yard of Fincantieri Marinette Marine, builders of the U.S navy's freedom class of literal combat ships and the constellation class guided missile frigates sustained a ransomware attack that disrupted shipyard operations. The U.S naval institute news reported the attack on Marinette Marine targeted servers that held data used to feed instructions to the shipyard's computer numerical control manufacturing machines, knocking them offline for several days. CNC enabled machines are the backbone of modern manufacturing, taking specifications developed with design software and sending instructions to devices like welders, cutters, bending machines, and other computer controlled tools. Fincantieri told the naval institute news that it had been working hard to remediate the problems, stating, "Fincantieri Marine Group experienced a cybersecurity incident last week that is causing a temporary disruption to certain computer systems on its network." The company's network security officials immediately isolated systems and reported the incident to relevant agencies and partners. Fincantieri Marine Group brought in additional resources to investigate and to restore full functionality to the affected systems as quickly as possible. Operations had begun to approach restoration within a day of the attack, but the episode was disturbing and was closely monitored by the U.S navy. It wasn't immediately clear whether any data was stolen, but if temporary disruption of construction at a shipyard was the attacker's goal then mission accomplished. The U.S intelligence community sees Russian cyber operations devoting more effort toward disruptions of supply chains supporting Ukraine. CyberScoop quotes NSA's Rob Joyce, the agency's director of cybersecurity, as saying that NSA is observing a significant amount of intelligence gathering in to western countries to include the U.S in that logistics supply chain. A significant fraction of that supply chain carries humanitarian aid, and it's worth noting that the supply chains at risk here are physical supply chains that move material goods, not the software supply chains that have been much talked about. This sort of threat has been seen before. To place some of the potential effects in context, recall the way that NotPetya, a pseudo ransomware campaign from 2017, encrypted systems at the shipping giant Maersk. The attackers continued their imposture enough to demand payment, but as it turned out there was no way to decrypt the files. They were modified in ways that made them unrecoverable. Loss of these administrative systems resulted in loss of visibility in to Maersk's containers and the shipper had to revert to manual checking and manual management of the cargo it carried. This caused supply chain delays particularly serious in time sensitive shipments. So reconnaissance should be taken seriously. A wise logistics organization will keep its shields up. Last week at the RSA conference in San Francisco a community of private sector companies announced the formation of ETHOS, an acronym for emerging threat open sharing. ETHOS is intended to be an open source vendor agnostic technology platform for sharing anonymous early warning threat information across industries with peers and governments. It's intended to function as a hot line across which early indications of threat activity can be shared. The 11 founding members of the ETHOS community are 1898 and Company, ABS Group clarity, Dragos, Forescout, NetRise, Network Perception, Nozomi Networks, Schneider Electric, Tenable, and Waterfall Security. The initiative also has the support of CISA, the U.S Cybersecurity and Infrastructure Security Agency. Eric Goldstein, executive assistant director for cybersecurity at CISA said, "The scale and threats facing critical infrastructure operators and in particular operational technology networks requires an approach to information sharing grounded in collaboration and interoperability. CISA is eager to continue support for community driven efforts to reduce silos that impede timely and effective information sharing. We look forward to collaborating with such communities including the ETHOS community to improve early warning and response to potential cyber threats while appropriately protecting sensitive information about our nation's critical infrastructure community." ETHOS is structured as a not for profit entity run by an independent mutual benefit corporation. At present its technology resources can be found on GitHub.

Dave Bittner: Our guest this week is Mike Hoffman, technical leader for global services at Dragos and a SANS instructor. Mike is discussing IT and OT misalignment. Mike, you and I have previously spoken about this notion of risk based vulnerability management within the OT space. And today I think we want to dig in to that a little farther, discuss the potential here for some IT and OT misalignment that can take place. Can you make your case here? What exactly are we talking about here today?

Mike Hoffman: Yeah. It's a really, really important topic especially when we think about when I look in a lot of the larger organizations today when they're -- you know people are still trying to figure out where OT fits. And we're -- and how IT can help, can bring in from the security sampling and really bring in their knowledge and their capability to help the OT space. And so a lot of different companies are still kind of going through this in a way that -- the identity crisis of OT and IT working together. One of the ways so that we see this alignment and/or misalignment is around the discussion around vulnerability management. From an IT perspective they have very, very mature processes and capabilities in place where they go out and they have, you know, either using end points or vulnerability scanners and other tooling to go out and assess their landscape, and then quickly, you know, determine and patch, come up with irrigations when they can, and those kinds of things to make sure that their systems are fairly up to date with some of the latest, you know, either OS or application patches. And again very, very solid workflows and processes. The problem, though, is that -- is that when we take that same mentality and we try to apply it to the OT side, things begin to fall apart. We just can't go out and take the same scanning capability or dumping end point, you know, pieces of software on our machines and to understand, you know, where we're at and then go out and blanket apply these patches. Reboot our machines at nighttime or we could go in the next day. There's this whole situation just doesn't quite copy over. And one of the challenges I see is that from the IT perspective is that this is their lever. This is one of the main things that they deal with risk. Simplified term of risk of course is, you know, risk equals the vulnerability. You know, and when we think about the vulnerability impacts and so forth it's one of the ways that IT folks can lever or manage risk is by affecting vulnerabilities and/or removing them. And so when we -- again when we take this and we apply it to the OT side, the challenges are is that some systems can be managed in this way. When we think about systems in the DMZ and others where a lot of those don't have to continually be up and running for our plants and systems to function. You can take a jump server as an example. You can patch it very, very frequently, and you should. And you can take a lot of these same processes, but when we go further down in the OT networks you cannot -- this recipe for, you know, mitigating and fixing vulnerabilities just begins to fall apart. And it's because of the way our systems have been designed. They're designed to function 24/7, although a lot of them haven't been really designed and they're getting better, but they're -- they don't have a lot of the capability or they're a lot more challenging to apply these patches. And plus the patches really need to be test embedded really before. So there's a lot of other things here that, yeah, it works, but this it does bring up a quite cultural clash if you will. So.

Dave Bittner: And how does this typically play out? I mean is it -- does it get to the point of being almost adversarial where the OT people are saying, "No. You cannot do this." And the IT people are saying, "We must do this."

Mike Hoffman: Yeah. It -- no. I mean it can be, but this is absolutely a time where it's a learning opportunity for both. For the OT side to understand how, you know, to help to automate some of it. And so this is where it's essentially -- and I've sat through countless meetings from, you know, working with IT vulnerability teams, you know, talking about, you know, the differences where some things can overlap, where we can have some alignment, where they can help us out in some ways. We think of certain industries where they're actually doing vulnerability scanning and I'm not advocating that you should go out and do this, but in certain industries they're absolutely doing this and they're taking a lot of the best practices from their IT counterparts and doing scanning in certain areas of the network. And it works well for certain organizations. And I'm not saying that this works for every organization, but some it does. And so also understanding how, you know, from the IT side making -- you know, making sure that we can leverage things like WSUS servers or other, you know, components where we can get qualified patches coming down and trickling down in to our environments, distribute them out, and again use some of that automation there that the IT side is very, very comfortable with using. Again, though, a lot of times we can't leverage a lot of those, the solutions, fully. We can't go, you know, reboot our systems at night like IT can. But yeah. We can learn from each other. We can learn and learn the capabilities. We can discuss the differences. So we both have that understanding. One of the things too is making sure that OT doesn't have the same KPI as IT. OT is always going to have some systems that, you know, don't -- haven't been patched or haven't been patched in quite a while. From the IT perspective if they're tasked with keeping track of OT vulnerability management they need to be aware and they need to be comfortable with some systems are going to be out of date. And that's just the way it is. And that's okay, but we also need to document other mitigations around them. Maybe we have -- that's in a section of the -- or the network that's been, you know, zoned off. We have additional network monitoring out there. We have different host based monitoring. So we're doing other things that we try to protect, but yet we are still able to detect. So we have the detective measures around it. Vulnerability management is a holistic thing and should be thought of in that way.

Dave Bittner: Looking at the long timeline and acknowledging that particularly on the OT side, you know, things can happen on a timeline of decades, do you think there's an inevitable convergence here?

Mike Hoffman: I do. So I mean we're seeing a lot of -- you know, so -- from a number of years ago when you talked about, you know, trying to bring in different technologies such as virtualization, a lot of people would absolutely mock it saying that my DCS -- how dare you say that my [inaudible] or DCS would be virtualized. And now we're seeing that that's pretty common. And so I think a lot of different solutions and technologies are coming out to help us in this perspective of keeping our systems up more where we're seeing better usage of redundancy. Again our systems have always been redundant for availability, but now we're looking at availability and redundancy from a patching perspective as well. There's also a lot of initiatives out there for the next version of what a DCS looks like. The next version of those kinds of things. And -- and some of that is going to more of this containerized based approach or making sure that we have systems that can be patched, you know, rebooted, spun over to a backup system without taking a plant offline. Without taking that controller offline that's manipulating the plant process. So I think that as our systems -- as we bring more of this technology here there will be, you know -- I think this will be easier. I still think that this is always going to be some sort of a challenge, especially when we think of discrete devices out there that there's always going to be that last end point device with firmware that you just can't -- you know, you just can't make that latest update before you have that maintenance window. So is it going to get better? Sure. But are we always going to be able to -- you know, is it totally going to converge where we have that same capability, that same rapid response? I don't think so. And I don't really think that that should be our goal either. We are always going to have systems that are designed in a certain way. Maybe not insecure by design, but yet there is -- there's always something else. And when we think about vulnerability management it's more than just patching. It's fixing a flaw or fixing a weakness. So if we can design our systems, design that weakness out, or shut that weakness off, if we can shut off a port or a service on a system, that that will in fact -- could correct that patch that we might not need to apply now. So we need to think about this as it's more than just applying a software patch or a piece of firmware. It's designing our systems out securely. It's making sure that we're building our system out in a defensible approach. So it's expanding our views and our horizons beyond just thinking about fixing this weakness.

Dave Bittner: In your experience with the folks that you've worked with with this, are there -- are there common elements to the organizations that are finding success here? Who are navigating these things successfully. Are there things that they have in common?

Mike Hoffman: I think so. And I think one of the most things that are in common are it's really working together. And so when you -- when you think about it it's leveraging the capability and the knowledge of both the engineering disciplines and also the IT security disciplines and those kinds of things. So we work best when we learn from each other, and that's really the commonality. I think another one too is the material organizations. A number of them for better or for worse they've been regulated. And so when we think about more it's they've been driven to a requirement. And I'm not saying regulation is good or bad, but it does -- it does push that topic of we have to achieve these certain requirements, these certain levels. So a lot of the more mature, you know -- again when I think about that I think about north America electrical sector where some of the [inaudible] regulations have driven. It's driven people to this requirement of, you know, we have to do this in a certain amount of frequency and time. And so it's driven by necessity doing this in a much more efficient way. And using a lot more of our capabilities. But electrical sector aside, when I think about a lot of the companies that I've worked with it's all about coming down, working together, leveraging best practices. And really learning from each other from that perspective.

Dave Bittner: Our thanks to Mike Hoffman from Dragos for joining us.

Dave Bittner: In this episode's learning lab Dragos' Mark Urban is joined by Dragos senior product manager Jordan Wilkerson to dig in to ICS network visibility and monitoring.

Mark Urban: I'm Mark Urban with another episode of learning lab here at "Control Loop." I want to show the attention to one of the five ICS cybersecurity critical controls as put out by SANS. Number three of those controls is ICS network visibility and monitoring. So that's definitely free from SANS. The system of systems nature of ICS drives a need for network monitoring to understand the interactions among those systems. ICS specific monitoring includes the [inaudible] and inspection of ICS protocols native to that environment. And it goes on. So I'm not going to read the entire description. That's from the SANS white paper. The five ICS cybersecurity critical controls. I encourage you all to go and read that. In fact, I'll put the link for that in the show notes. But as we turn to network monitoring this is an area that is referenced in emerging rules and existing regulations or emerging regulations, existing regulations and frameworks. For instance Ferc directed NERC to develop requirements for electrical utilities to implement internal network security monitoring or INSM. The TSA regulations for pipelines include requirements to provide continuous monitoring and detection. So this concept is not a new one, and I think the SANS paper highlights it as a critical control. And I wanted to spend a couple of episodes on that, and the first of those episodes is focused really on the starting [inaudible] which is understanding kind of the systems within your environment sometimes known as asset inventory or often known as asset inventory. And for that I'm joined by senior product manager Jordan Wilkinson here at Dragos, and Jordan works a lot with organizations/industrials focused on security. Welcome, Jordan.

Jordan Wilkinson: Yeah. So first thank you so much for having me on. I really appreciate it. So yeah. So when we're talking about building out an OT security program there are generally four pillars that we discuss. Right? First and foremost -- and I think the topic of this conversation that we're going to have is around visibility in to your environment. Right? Being able to identify, A, what you have out of all those things. What are the most critical, you know, assets in your environment? And being able to evaluate sort of changes to that environment dynamically. So that's number one. And once you've established that visibility, then the next step is, okay, well out of all the things, which of those are vulnerable? And what vulnerabilities are on there with context around why they're important or not within the context of my environment? Right? So it goes from visibility in to viewing vulnerabilities in the environment and how you're going to manage those. So we really just want to provide that transparency so that you can decide how you want to view and manage context within your environment. The next is how do you then continually improve the level of threat detection in the environment. How do you see unauthorized traffic between IT and OT as an example? Can you detect adversary behaviors? Right? And as new knowledge comes out in the ecosystem, how do you get that in to your environment? And then, you know, event handling, right, as well is something that's critical. So something happens. You see these things occurring in your environment. And how are you going to manage those to make sure that they're remediated? Right? And so generally it's just incident response to those kinds of things. How can you operate using whatever tools you have to be able to manage those events?

Mark Urban: Got you. So this ability to ask, that's one, you know, detect and manage vulnerabilities, two, threat detection three, and then managing those incidents and events, number four, is kind of some of the foundational elements that you see. That's a great summary. Maybe we can take -- and I think among those I think we want to focus this discussion right now on that first one, especially visibility in to assets or, you know, when you -- when, you know -- asset inventory I think is, you know, probably a -- one of the most common terms in this. Can you help us kind of understand that world? What is an asset inventory? And we'll go from there.

Jordan Wilkerson: Yeah. Yeah. So I'm a bit long winded. So cut me off whenever you feel the need, but I'll try to get through this very quickly because, you know, I do think it is a general term that's used a lot, but I also think it's a bit subjective based on the biases of whoever's answering the question. Right? If I work for a vendor that provides X, Y, and Z data, of course I'm going to tell you, you know, the answer is a platform that provides X, Y, and Z data. Right? So I'm not going to do that. I want to -- I want to talk generically about this as I see it. Right? In my interactions with customers and what they care about. So in my view an asset inventory is a dynamic repository of data about physical assets that are on and off your network in your OT environment. And they're what I'll call different categories of attributes. Right? So I think about asset attributes in three general categories. And I can define them if you'd like. So it's basically the way I think about it is their direct attributes, their indirect corresponding attributes, and then customized attributes. So I do think it makes sense to kind of run through those, if that's cool with you.

Mark Urban: Yeah. I -- absolutely understanding what you mean by those I think would be key. Sure.

Jordan Wilkerson: Yeah. So let's talk about their direct attributes first. So this is information about the devices themselves. What they physically are. Right? What's their vendor, their make, their model, their type, their classification, etcetera? Information about what's running on those devices potentially. Right? Like their operating system, various configurations, network information. What's their MAC address? What's their IP? Or if they don't have one, have they ever had one? And if so, what was it? Right? Their last seen IP. That kind of thing. Also where are they physically or logically in your network? And then of course you can go even further and say, you know, "What additional software is on these devices, etcetera?" Then you have sort of the secondary category of attributes that I call indirect corresponding attributes that people might snuff at that term, but these are things like how are the assets behaving and communicating with one another. What are they talking to? Over what protocol? What is the directionality of those communications over what ports, etcetera? Right? And then of course, just like we just mentioned if there are vulnerabilities associated with those assets, those physical assets, what are they? Right? Which ones have been identified and what's their severity and information about the -- the vulnerability that in my mind I think of that as a discrete object, like a separate thing, but it's associated to an asset so it's indirect? And so I classify it like that. Likewise, you know, you have notifications or events that fire. Right? In most OT security platforms. And those are going to hit assets. Right? Assets are what notifications fire because of a behavior change or something like that. And so you want those associations also there. And the last category is probably the easiest one, and that is customized attributes. And that is basically anything that is input either by a software solution or an actual person. Right? Who knows more detailed information that maybe you can't pick up on the wire. Right? Like the criticality of an asset is subjective. Right? Its Purdue level isn't something that inherently isn't going to be detected. Customized tags. Customized attributes. Things that are pertinent to you or your organization that help you identify and manage your environment.

Mark Urban: There's a good description, you know, kind of what they are. The second is kind of what they behave in other contexts about it, and third is, you know, additional information that really, you know, develops a profile of this specific asset. And that makes sense. And to your point, the asset is the primary kind of construct in OT security to focus on because a lot of things pivot off that whether it's vulnerabilities or, you know, if there's an event that's typically tied to communication to or from an asset or the behavior of an asset. I like that. So right? Asset inventory with some characteristics why it's important. And give us a flavor for some of the characteristics of developing an asset and managing that over time like which people, you know, strive to do or some of the complications and thins like that.

Jordan Wilkerson: Well, there's a couple things. So if you remember in the beginning of my sort of long winded response of like what is it, right, I mentioned that it's a dynamic repository. Right? That's the opposite of static. Right? It's ideally hopefully god willing it's not an Excel document. You know, which is -- it can happen. You environment's ever changing. Right? It's ever evolving. And so, you know, when I think about building one out, it has to be able to adapt and evolve over time and show the disparity between what you had before and what you have now. Right? And the fact that, you know, you want it to be a platform that's flexible, that respects your own knowledge. Right? There's no software solution that generates information that is always going to get it right. And there's no vendor that's going to know your environment better than you do. And so the way I look at this is before I kind of get in to the importance here it's really about enabling you. It's augmenting your ability to say, "Do I agree with this or not?" And have the ability to override is probably a harsh term, but agree or disagree with the data that you're seeing and be able to effectively modify that information so that it makes sense to you. So now getting in to kind of the -- and to do so without having to have the platform or whatever it is that you're working with to manage your asset inventory then re-override what you just overwrote. Right? You don't want to wrestle with the solution. So why is this all important? Well, I think the general phrase that I'm sure everyone's going to cringe at when they hear it because it's said ad nauseam is that you can't protect what you don't know about. Unfortunately even though it's said all the time, it's true. So when it comes to securing anything, especially in an OT environment, you need to be able to make solid decisions. You need to be armed with critical information and as much information as possible. And most importantly you want context about that data. That could be about threats in your environment, about the vulnerabilities that you're seeing in the environment, etcetera. And ideally you want a solution that's going to provide you guidance on what to do within the context of your environment when you see those things.

Mark Urban: I'm going to interrupt and pull you off kind of the -- kind of the outline you probably are looking at to track along. You mentioned a couple of things that -- like you mentioned earlier in the discussion that the IP address that you were last seen at or you were saying that the inventory is a very dynamic thing that's not, you know -- you don't want to manage off an Excel spreadsheet. Are the changes that dynamic in the environment? I mean I thought these were like, you know, static industrial systems that don't change. You know, how -- you know, give us a little flavor for what you're talking about as far as the dynamic nature of kind of that environment.

Jordan Wilkerson: It changes all the time. Right? IP addresses just as a simple example. Let's stick with that. Right? IP and MAC addresses. Address association in general. These things can bounce around all the time. Right? And sometimes you'll have assets in your environment, particularly in OT, that are doing specific things. Right? And they don't necessarily have to do those specific things all the time. But just because they're not there doesn't mean you don't have to know about them. So when they go off network you need to be able to know that they still exist and you need to be able to reference historical notifications that are fired on those assets even though they're not actively communicating at any given point. So if you're using dynamic address allocation and you have an asset that falls away, that IP could shift on to another asset. Right? So things can bounce around. And then you have switches and all other kinds of traditional networking stuff, and that gets mixed in there. And you've got all kinds of craziness that can happen. So you really just want to be able to actively understand the current state always first, but you still need the ability to reference sort of the historical let's call it legacy view of your environment as well.

Mark Urban: The asset is probably still there, but it's not talking or it's been disconnected or it might be silent or it might have a different address. Even if it was active one week, three weeks later it might be active, but under some different addressing schemes. It's a dynamic nature because components turn on and off or, you know, are talking sometimes and not talking sometimes. And switch, you know, IP addresses. And okay. That makes sense. That gives a little bit more insight in to the dynamism of that. I mentioned that, you know, some environments are much more dynamic and some might be a little bit more static, but you always want to keep on top of those, you know, the current state or point, but also what have you seen in the past in case that factors in as well. Okay.

Jordan Wilkerson: 100%. Yep.

Mark Urban: Okay. So are there any other kind of complications that you work with with kind of industrial organizations in kind of that asset inventory realm?

Jordan Wilkerson: Well, I think it's not anything that is sort of unique to my experience I don't think. I think everybody that's in OT kind of wrestles with the same things. And one of those is just the primary difference between sort of traditional thinking of how you identify assets in general. You know, like in an IT environment you can go and you can run active scans and get responses from all of those kinds of devices because they're meant to be seen, share data, etcetera. In an OT environment that is not necessarily the case. In fact like you can take down critical assets by running active scans in that environment. So you have to do this passively and because you have to do that passively I mean you're just picking up traffic on the wire and dissecting these packets and making them make sense to an end user somehow. You have to be really clever on where you're capturing that traffic. So the issues that I've seen during my time at Dragos has really been a couple things. One is lack of protocol support. Right? Because you're dealing in an environment that has many, many, many dozens, hundreds, potentially of proprietary protocols. And those protocols don't necessarily exist in every single industry and every single kind of deployment and every single OT environment. So you can have missed protocols where you're just mischaracterizing things. Maybe you don't support a protocol at all so you don't see that traffic. It could also be that, you know, you have air gapped or enclaved areas of the network that your sensor can't see in to, right, that maybe somebody generically would expect to see. Those kinds of things are always difficult. But, you know, the risk of just being able to actively go in to an environment and scan everything and hit everything, just, you know, pump out like a big -- a big old network scan is too dangerous and too risky to do. So that's the trade off there is that generally it's really around mischaracterization, directionality issues, and lack of protocol support that I've seen are the big ones.

Mark Urban: Aren't people saying like, "Oh, where's this big set of stuff?" And why isn't that going up? And then they're like, "Oh yeah. It's -- that's an air gap. Oh. I forgot. That's an air gap network. Okay. There's --"

Jordan Wilkerson: Not to mention that the connections can be very low bandwidth. Right? So it's, you know -- that also presents issues too. So these are geographically dispersed environments usually. And even if they're single sites, they're complex, man. So, you know, it's hard to get all that stuff right all the time.

Mark Urban: You know it's funny. We -- the prior episode we had Rob Lee on talk about, you know, some of the differences and, you know, a little bit about convergence of OT and IT. But, you know, you reminded me like these OT environments are like networking, you know, lands back in to the '90s. Right? That that's, you know -- if you were like in to an IT environment, you know, it's probably 90% you know between HTTP and DNS. It's probably 90% of an IT network. And that's maybe a little bit extreme on the percentages, but in industrial networks to your point like the number of protocols and the diversity of communication types is just crazy. And I think that's one of the things that, you know -- the devices are different. Right? You're talking about -- when you're talking about OT devices you're talking about what? TLCs and, you know, distributed control systems.

Jordan Wilkerson: DCS. Yep.

Mark Urban: Plant controllers, you know, in manufacturing, and all these things. And their discussions that, you know, those -- you know those secondary attributes that you were talking about about how they talk is just so completely different because these are often proprietary, you know, equipment OEM vendors with their proprietary protocols speaking proprietary commands and things like that that don't look anything like, you know, HTTP access [inaudible] the IT world. And it's -- it's interesting that, you know, you talked a lot about the devices. And I think you really hit on something that, you know, I just wanted to emphasize. It's just the incredible complexity and diversity of those discussions and the different nature of those devices in those environments as well.

Jordan Wilkerson: IT and OT and vice versa, you know, and I would -- I would love to be able to like, you know, family feud it and be like go me my OT inventory. Right? But it's not that easy, and you know Steve Harvey's probably very expensive. Anyway it's not -- it's not an easy thing. The good news, though, is that every deployment we're learning, right, and every time that we go in and we find out, oh, we don't support X, or oh -- or mischaracterizing something or what have you, we have mechanisms in place to be able to manage those things. Right? This whole OT world is pretty good at like sort of general community defense and so, you know, we -- we feed off of each other and our knowledge base continues to grow so that let's just genericize it and say industry to industry over time similar deployments become easier. They never get easy, but they do get easier. Right? But it -- but every environment is different. Every environment's super complex. So it's definitely a challenge.

Mark Urban: Jordan Wilkerson, everybody. A little bit about asset inventories, managing all that communications. Jordan, thank you so much today.

Jordan Wilkerson: Thank you.

Dave Bittner: And that's "Control Loop" brought to you by the CyberWire and powered by Dragos. For links to all of today's stories, check out our show notes at thecyberwire.com. Sound design for the show is done by Elliott Peltzman with mixing by Tre Hester. Our senior producer is Jennifer Eiben. Our Dragos producers are Joanne Rasch and Mark Urban. Our executive editor is Peter Kilpe and I'm Dave Bittner. Thanks for listening. We'll see you back here next time.