Looking to the future of the OT space.

Dave Bittner: It's November 16, 2022, and you are listening to "Control Loop." In today's OT cybersecurity briefing, the U.S. Department of Energy seeks to improve visibility into ICS environments. NIST has issued a proposal for upgrading cybersecurity at water plants in the U.S. A patch has been issued for a critical vulnerability that affects flow computers from ABB. Our guest, Ashif Samnani from Cenovus, shares lessons learned from nearly two decades of OT experience. Today's Learning Lab is the third in a series with Mike Hoffman, principal industrial consultant at Dragos, teaching infosec professionals how to think about OT security.

Dave Bittner: Representatives of NATO's member countries met in Rome last week on Wednesday and Thursday to review and renew the Atlantic Alliance's Cyber Defense Pledge. Most of the proceedings have been closed to the public, but the U.S. State Department announced ahead of the sessions that cybersecurity for the energy sector is figuring prominently on the agenda. And some of the second day's keynotes are publicly available. We sat in virtually on one of them by U.S. Deputy National Security Adviser Anne Neuberger. She began by noting that NATO has remained relevant as the world and the technology in it have evolved. She stressed that recent experience, especially what she describes as Russia's brutal and illegal war against Ukraine, has highlighted the importance of cybersecurity preparedness and partnership. This is not only a war-time lesson. 

Dave Bittner: As Neuberger went on to say, effective international partnerships are necessary to defend against transnational threats, and such threats are endemic in cyberspace. Preparation and partnership seem to have spared Ukraine much of the worst that had been expected of Russian cyberattacks. Russia's hybrid aggression against its neighbor has turned, in recent weeks, to attacks on critical infrastructure, especially basic civilian infrastructure like power and water distribution. The Russian attacks have been directly and violently kinetic, delivering high explosive and not the malware packages that had been widely expected during the run-up to war. This suggests that Russia, whatever its intended aims were, is no longer planning on having to restore basic services in conquered provinces quickly, probably because Moscow has begun to doubt the full victory it expected until it was ejected from the towns around Kyiv. It's also a sad reminder that resilience must be prepared against a full range of eventualities from DDoS, through severe weather and all the way to sabotage and missile strikes. 

Dave Bittner: The U.S. Department of Energy is adjusting its security strategy to increase OT threat monitoring, according to GovernmentCIO magazine. At AFCEA's Energy, Infrastructure and Environment recent summit, the DOE's chief information officer, Ann Dunkin, said that increased connectivity with smart devices could have unknown consequences. Dunkin stated, every one of those entry points that you and I have and the utility companies have is an entry point for cybercriminals to break in. We have to secure not just, you know, the operation center, but your house and your car and the batteries on your house and your thermostat. Those are connected to the grid. And the last thing we want to hear is that someone's Nest thermostat brought the grid down in D.C. 

Dave Bittner: Additionally, Puesh Kumar, director of the DOE's Office of Cybersecurity, Energy Security, Emergency Response, stated that the department is teaming up with the private sector to improve monitoring of ICS environments. Kumar said, for a very long time, we've done a really good job on the IT. side. OT is where we're seeing a lot of cyber-adversaries focus, and that's the part of the network that can actually have impacts on energy delivery. We really need to get visibility. Ann and her team are thinking about that visibility into the PMA networks. We're thinking about it and working with the energy sector to also deploy similar technologies. 

Dave Bittner: GovCIO notes that last year's Infrastructure Investment and Jobs Act handed the Department of Energy $62 billion to make improvements to the power sector. Kumar stated, this is a strategic opportunity like we've never had before. In many ways, with the grid of the past, we were bolting on cybersecurity. I think we have an opportunity now that we can all be seizing upon where we actually design the grid with cyber-informed engineering, where engineering things more securely from the get-go. 

Dave Bittner: FedScoop reports that the National Institute of Standards and Technology is soliciting comments for a proposal that would improve cybersecurity for the water and wastewater sector in the U.S. The project is being run through the National Cybersecurity Cenmter of Excellence. NIST is accepting comments until December 19, 2022. The NCCOE is seeking input from water utilities of all sizes - small, medium and large. The center stated in its announcement the increasing adoption of network-enabled technologies by the sector merits the development of best practices, guidance and solutions to ensure that the cybersecurity posture of facilities is safeguarded. The NCCOE will demonstrate use of existing commercially available products to mitigate and manage these risks. The findings can be used as a starting point by utilities in mitigating cybersecurity risks for their specific production environment. This project will result in a freely available NIST cybersecurity practice guide. The NCCOE added that many OT devices are now converging upon information technology capability with the advent of industrial Internet of Things devices and platforms, such as cloud-based SCADA and smart monitoring. 

Dave Bittner: Are water utilities are at risk? Sure, they are. We're seeing in the hybrid war against Ukraine right now that kinetic attack often succeeds or replaces cyberattack. And as we mentioned earlier, water has, with electrical power, been the infrastructure sector, most targeted by Russian missiles this past month. 

Dave Bittner: Claroty has discovered a high-severity vulnerability, CVE-2022-0902, affecting flow computers from electrical equipment provider ABB. These computers are used to measure oil and gas volume and flow rates. And Claroty notes that these measurements are critical not only to process safety, but are also used as inputs in other areas, including billing. Claroty responsibly disclosed the vulnerability, and ABB has issued a patch. ABB stated an attacker could try to exploit the vulnerability by creating a specially crafted message and sending the message to an affected system node. This would require that the attacker has access to the system network by connecting to the network either directly or through a wrongly configured or penetrated firewall, or that he installs malicious software on a system node or otherwise infects the network with malicious software. In general, an attacker could brute force its way past authentication, and from there, use a path traversal vulnerability to achieve root access. So far, ABB says, according to SecurityWeek, that it's not aware of any exploitation of the issue. 

Dave Bittner: Finally, on the eve of Veterans Day last week, CISA released 20 new industrial control system advisories. They affect 16 Siemens products, two Omron products, and one product each from Delta Electronics and LS. Go to CISA's website and find the notes on the system you use. 

Dave Bittner: Our guest, Ashif Samnani, is industrial control systems cybersecurity leader at Cenovus Energy. He has nearly two decades of experience in the OT world, and he shares with us these insights. 

Ashif Samnani: I'd say I've been working in the cybersecurity space for the last 17 years. I've worked for various industries, including upstream, downstream oil and gas, technology and finance. I've worked for companies such as Symantec, Spectra Energy, Enbridge, Husky Energy and Cenovus Energy. So I've worked in various aspects of cybersecurity, including threat management, vulnerability management, incident response, forensics, governance, risk and compliance and industrial control systems, cybersecurity. 

Dave Bittner: And in the nearly two decades that you've been in the space, I mean, what are some of the real significant changes that you've seen? 

Ashif Samnani: Within the OT side, I've seen automation of discovery of new vulnerabilities and threats within the environment. The technology has been evolving. So what we've been doing in the IT space is similar to what we're not doing in the OT space, right? So there has been an involvement in the types of technologies we've seen. Even the evolution of threats within the space has become far more apparent, right? I remember back in 2012, I was doing some research around Stuxnet. That was one of the first significant cybersecurity threats within the OT space. And now we see quite a bit relative to the OT area, nothing as prominent as Stuxnet, but we've seen quite a bit, right? So it's just an evolving space within the OT and ICS area of cybersecurity. 

Dave Bittner: Yeah. I mean, as you look at some of the threats that the various OT organizations are facing here, does any in particular stand out to you in terms of the potential vulnerability? 

Ashif Samnani: Well, the biggest vulnerabilities - and this is what we see in the OT space - is that a lot of OT incidents are - stem from the IT space, right? So what we see is a lack of segregation within the OT environment or even proper segregation. So that's one of the things that we still face within organizations. So my team is responsible for the segregation between the IT and the OT network. And we're looking at a complete segregation model where we segregate only - not only at the network layer, but also at the identity layer, right? So we're trying to complete full segregation within our organization. I'd say we've done quite a bit. I'd say about 75 to 80% has been completed. We're working on our last stretch so we can complete the segregation model. 

Dave Bittner: Can you give us some examples of some of the specific challenges that you and your team have faced when trying to go through something like this? 

Ashif Samnani: I'd say we haven't seen too many challenges. I'd say when we first initially did this at Husky Energy, because Husky actually merged with Cenovus Energy, one of the things around business adoption and getting buy-in from the business so we can conduct our security assessments. The segregation I didn't see much of an issue with, but as we started going into the OT environment, I feel that the business didn't have too much confidence in terms of what technologies are out there to support the cybersecurity objectives within the OT environment. So I'd say that business adoption was one of the biggest challenges that we faced. 

Dave Bittner: So from your experience, are there particular OT verticals that have a better time with this journey than others? In other words, you know, does water treatment plants do better than electricity or vice versa? 

Ashif Samnani: I would say we see verticals that have done quite well, actually, is around the electricity verticals, right? So especially around NERC (inaudible) requirements. I think we've seen quite a bit there in advancement. The oil and gas sector now is now seeing significant, you know, adoption rate as a result of regulatory requirements such as CFATS and the U.S. Coast Guard requirements. So we're seeing a lot of new cybersecurity requirements, even TSA, whereby we actually have to implement cybersecurity controls. So now as the regulatory requirements are starting to come into play, there is a stronger adoption rate as it relates to implementing cybersecurity controls. 

Dave Bittner: When you look at a potential timeline for getting folks where they need to be, what do you suppose is realistic? Are we talking about years, decades? Where do you suppose we stand? 

Ashif Samnani: I'd say our first journey took about - it depends on the size of the organization. So, for instance, when I was at Husky Energy, we had a total of a hundred sites, a little bit over a hundred sites, large refineries and stuff. With our journey, it took roughly about three years to complete the network segregation, because we also have to consider turnaround times within the different facilities, right? So I'd say three years for your initial program and then another three years for a second round if you want to build like OT network visibility as it relates to ICS cybersecurity - right? - and vulnerability management, rate? So if we split the cybersecurity program into two, I'd say anywhere from five to six years for a company this large. 

Dave Bittner: And what do you suppose some of the roadblocks are for people to put in place the things that need to be done? 

Ashif Samnani: I'd say some of the roadblocks - there are always technical roadblocks. For instance, having old and archaic technologies within the OT environment is one of the challenges. So using traditional tools can be difficult, right? That's some of the technical roadblocks. Timing - our facilities run 24x7, so we need to work with the facilities to ensure that we plan for turnarounds and implement our cybersecurity controls then. I one of the challenges also is around standardization of technologies, because especially with large oil and gas companies such as like Husky or Cenovus, we tend to work in a federated model, right? So there's always a challenge with implementing different technologies because of the technologies that are currently implemented, whether it's Rockwell, ABV, Siemens, any of those technologies. Some of them don't fare well with certain types of cybersecurity technology, such as antivirus. Some may use Symantec. Others may use McAfee. We always have had challenges deploying EDR technology such as CrowdStrike because it hasn't been fully compatible with some of our ICS technologies. So there are some roadblocks there, right? Sometimes the technology is not mature enough to handle archaic OT spaces. 

Dave Bittner: I'm curious. You know, it's practically a cliche that there's, you know, tension between the IT and the OT sides of the house. I'm wondering, in your experience, how accurate that is. I mean, we've gotten to the point where teams are getting past that? 

Dave Bittner: We're evolving now because the IT and OT space is slowly starting to converge. I'd say let's flip back to 2012, when I first did OT cybersecurity. There was a large disconnect between the organizations, right? Between the IT and OT space when I worked at Spectra Energy, right? The business was not adopting best practices that IT dictates. Plus, you also have the mindset of an IT person going into an OT space. Typically OT personnel are engineers. They understand that technologies a little bit better. But nowadays - right? - you're seeing the IT and OT teams working very closely because they understand that IT - OT threats are primarily stemmed from IT-specific incidents, right? 

Dave Bittner: So we're seeing tremendous adoption, especially the fact that, like I said, new regulatory requirements are coming into place. So we need to ensure that the OT space is secured and they're working very closely with IT. So regulatory requirements really drive a lot of the spaces, plus also the known incidents, for instance, like Colonial that resonated with the OT groups. And they were concerned about their security posture. So they're working closely with the IT teams and stuff, right? I know at the current company, we work very closely with the various teams within the OT space, so we don't see much of an issue these days. But if we flashback like five to six years ago or even 10 years, yes, there was a significant issue in terms of working with the IT group. 

Dave Bittner: And what about management, you know, the powers that be? Where are they on their journey of kind of understanding the resources and tools that folks like you need to accomplish your mission? 

Ashif Samnani: I would say they get it now, right? But it's always reactive - right? - and that's a challenge - right? - that we face when a large cyber incident such as Colonial comes into play. It's like, wow, OK, we need to do this, right? Proactiveness - it's there, right? I'm not saying it's not there. But to really get management - typically and it's always been this case where you stumble into large industry based cyber incident - right? - and that's a wake-up call always. But I think management needs to be a lot more proactive, right? I've seen it in various companies where it's a little quiet until something happens, right? And that's still the case. 

Dave Bittner: Where do you suppose we're headed here? As you look towards the next few years, any notions for how things are going to evolve? 

Ashif Samnani: Yeah. I mean, I could speak a few, right? For instance, in the OT space, the adoption - and this is already happening - is adoption of cloud - right? - within the OT space. That's one of the things that we're facing, especially with companies such as AWS that are building like specific data lakes related to data historians - right? - which is not commonly found. So now what's happening is where - the boundaries of OT, they're changing, right? We're not only going into the IT network, but we're going to the cloud, right? So that's a adoption - right? - that I see. In addition, the new technologies which are coming out that leverages AI and machine learning to detect for threats and vulnerabilities, we've seen a lot of those coming up. But I think that's growing, right? The threat and vulnerability platforms are evolving also, right? So maybe next generation, like, threat management systems coming into play, which fare better in the OT space. Typically, technologies right now based off of the architecture, they don't fare well. Sometimes we don't have that complete visibility. But I think we'll see - find better technologies within the space. 

Dave Bittner: Are you optimistic that we're going to get there, that we'll get a good handle on these things? 

Ashif Samnani: I'm very optimistic, right? I've seen this industry grow over the last 10 years, specifically the OT area. I think we'll get there, right? And as regulatory requirements come into play, another one I forgot to mention was Bill C-26, which is in Canada - right? - that takes cybersecurity requirements for critical infrastructure - companies that employ critical infrastructure, right? So I feel heavily confident that we will get there, right? It'll take a little bit of time, but I'm sure with executives understanding the new requirements from a compliance standpoint and the evolving threat landscape, they'll take this a lot more seriously and consider the investment. 

Dave Bittner: Our thanks to Ashif Samnani from Cenovus for joining us. 

Dave Bittner: In today's Learning Lab, we feature the third in a series with Mike Hoffman, principal industrial consultant at Dragos, teaching InfoSec professionals how to think about OT security. Here's Mike Hoffman. 

Mike Hoffman: This is something that I do a lot when I'm dealing with customers and we're on an engagement is looking at crown jewel analysis. And this is really understanding what really matters within your environment. And it's understanding those key areas that you have to have operational. And again, this is digging into operations. So when you come from that IT perspective, it's really important to sit down with your operations folks to really understand, the security controls we're putting in place, what are we actually safeguarding against? What physical systems out there do we have in place? And what are the systems that are most important? Because again, not all ICS devices are the same. And each one of them may have different levels of process impact. 

Mike Hoffman: When we do identify those areas, we probably need to be thinking about other mitigative controls to put in place and - or mitigative controls, engineering controls, you know, again, even maybe engineering it out. But going through the CGA process is not something that your IT security folks can do. It's not something that your OT security folks can be doing in a vacuum. This requires a team. This requires folks from operations and so forth to really sit down and understand your process environments. And I'll first of all say that - probably have a lot of crown jewels within your environment, probably more than one. So this is a methodology, if you will, that you can take back into your companies. And you can begin to work these processes out. 

Mike Hoffman: But first of all, what you do is identify the top-level systems or - and you - and so you're looking at the regions, the demographics and so forth. You begin to get into the subsystems where you're looking at, you know, the collection of facilities and the environment. And then you get actually down into the critical functions. So what is the areas - you know, this may be utilities. You may be hitting steam and that kind of thing. And then you look at the critical components. What makes up that critical function? What's the components underneath? These may be all your pumps and motors, you know, your actual boilers and those kind of things. 

Mike Hoffman: And then we get down into the controllers. These are the devices that are actually, you know, is - this is that interface from logical into physical. And these are those systems that, if they are compromised, attacked, manipulated, can actually cause some sort of a effect to your your process environment, your plant floor, your systems. And then we kind of work back up the pyramid, if you will. And once we've identified your crown jewels to say, what - so now we've identified the things that can actually be impacted. Let's work back up this pyramid to understand how the system is interconnected with other systems. We may be able to see that, you know, we have a lot of connectivity around this critical system that we may want to put some more security controls around. We may want to monitor. We may even want to isolate this. This thing has been deemed so critical and important that we actually need to disconnect it, maybe hook it up with a serial connection. You have to go backwards in technology, but sometimes it's a requirement because this thing cannot - absolutely not be compromised by a RATable protocol. So we may actually even have to isolate it with a, you know, think about RS-232 and 45 that might need to occur because, you know, due to the risk of this system. 

Mike Hoffman: One example of what I want to share here really quick is that I found - working, again, in a refinery environment - I've seen refineries kind of fall on their face, if you will, trip offline a number of times. And one of the things that did it, actually, was plant error. And you think about that. And so we can take this analogy, though, into any environment that requires instrument error. So instrument error drives control valves. It drives different - those physical systems within the process environment. So control valves need errors. So let's talk about the refinery for - really quick. So what is the system? Where does air come from? It's in the boiler house or whatever you would call your utilities area. Then we dig down into that critical function or subfunction. So we begin to identify instrument error. You'd normally have some sort of pressure control system. You're also drawing your air. So you have dew point. You have dew point analyzers and so forth. And you have systems around your critical function. Your components, well, of course, your compressor. So you probably have a couple of different air compressors out there. You have your pressure control station and, of course, your dryers. 

Mike Hoffman: When we look into your controllers, though, you may have a vibration monitor around that compressor. That vibration monitor probably has shutdown settings on it. So you need to look at, OK, so that thing can actually trip my compressor offline. Normally, you have some sort of a DCS system that's controlling all of your pressure controls around your instrument error. You have your dryer PLC. So you have, you know, some sort of a drying mechanism to dry your air to get the moisture out of it. And then, of course, your do dew point analyzer, which is actually measuring the dryness of that air. So then when we get down into the crown jewels, we begin to say, OK, so what we really need to do is be protecting this vibration monitor. We need to be looking at the interconnections, if you will, of that system. Can somebody remotely get to it, change settings? Look at the PLC. Can somebody get to this PLC from the network? You know, what kind of controls do we have around that, on the DCS system, the analyzer? And we kind of go back up and look at the remote access capabilities, look at our firewalls, look at those rules. So again, this is just an example of crown jewel analysis. But it's - but I think it's a very, very - an easy one to kind of walk through. And so you can kind of take this back into your work - place of work and see how this can actually be used in your environment. 

Mike Hoffman: When I think about security and where do we start - so we begin with architecture. And architecture is really where you get the most bang from your buck, if you will. This is where we are ensuring that our systems are - we're all architected out, and we're also maintaining our systems. I'll be talking about architecture here in a little bit around the actual areas, but then we get into passive defenses. These are systems that can work without the human involved. Think about things such as, you know, monitoring tools, your firewalls and so forth, that are all there to passively defense. And then we get into that active defense. And this is where we have people in our environment. They're actually interacting with our environment. These are your analysts, SOC analysts, and those kind of things that are looking through the logs, looking through that network traffic and getting into those phases. As you move over into intelligence, your maturity's coming up. This isn't creating your intelligence. This is often consuming your - the intelligence. Could be from ISACS, could be from other areas where you're getting intelligence down. 

Mike Hoffman: So with that - focused efforts. So now let's get into some of the discussion around, how do we - where do we start? Because the issue of securing OT is big. It's challenging. And so a lot of times, I've heard - and I've been in a lot of discussions around, you know, trying to map out, put in all these controls, trying to, you know, align different controls to different frameworks and so forth. And that's all well and good. But a lot of times, we get - we tackle this from too much. We try to do too much at once. And security programs are left kind of faltering and failing because we're trying to bite off too much. We're trying to get from crawling to running, and we haven't gone through that kind of getting up and walking slowly, if you will. 

Mike Hoffman: Some of the best organizations that I've seen that are doing this are working together. It's forming that cross-functional team with multi-backgrounds - from the IT, from the OT security, bringing in folks from operations, bringing in folks from control engineering and so forth - and having those discussions, and also realizing that this is absolutely a journey and it's not a project. The first thing that we do is doing an architecture review and doing that crown jewel analysis - understanding the important aspects of that environment. The next thing we do is we get into data hunting and collections and those kind of things. We may, at this point, actually do maturity-type assessments to understand how well your environments are and how well your process are in place - moving into thinking about, you know, working through tabletop exercises and so forth. So once you've understood your environment, you can begin to do tabletops. 

Mike Hoffman: You may be thinking about, you know, doing penetration testing. Well, you know, it's always kind of that cool thing to do, but penetration testing should only be really done after you've done the basics, after you know you have a fairly good architecture, you've done everything you think is possible. Now let's test it. Let's make sure maybe from a assumed breach type of a penetration test that our systems can actually withstand that or we can actually detect that, what's going on. That's a great time to do a pen test and also get into your systems and do a managed threat hunt. 

Mike Hoffman: At the end of the day, though, yeah, IT, OT is absolutely different. Even though we are utilizing the same types of systems and a lot - and the higher level of our networks in our environment, it is different because - and it really depends on the context and how we're using our IT systems. Defense is doable. Protecting our systems is doable. It's hard, but it's absolutely doable. And you have an absolutely important role to play in this. So no matter what your background, if you are doing something from an - from - if you are working somehow in the OT space, your role is incredibly important. And thank you for what you do because you're helping to safeguard your companies and ultimately safeguard and provide products and services to civilization. 

Dave Bittner: And that's "Control Loop," brought to you by the CyberWire and powered by Dragos. For links to all of today's stories, check out our show notes at thecyberwire.com. Sound design for this show is by Elliott Peltzman with mixing by Tre Hester. Our senior producer is Jennifer Eiben. Our Dragos producers are Joanne Rasch and Mark Urban. Our executive editor is Peter Kilpe. And I'm Dave Bittner. Thanks for listening.