The fundamentals of the control loop.

Dave Bittner: It's September 9, 2022, and you're listening to "Control Loop." In today's OT cybersecurity briefing, we hear about a new study of cybersecurity in the food industry. Montenegro works to recover from a Russian cyber offensive. NSTAC recommends cataloging federal OT assets. The U.S. administration is turning regulatory and policy attention to chemical sector cybersecurity. Kinetic attacks affect a Ukrainian power plant and serve as a cautionary tale for cybersecurity. And CISA issues more ICS alerts. Our guest is Dean Parsons from the SANS Institute. He'll talk us through the cyberattacks against critical infrastructure, the future of geopolitical conflict, active ICS defense using ICS threat-hunting techniques and the difference between IT and OT incident response. And in the Learning Lab, the story of how a toilet became the first industrial control system more than 2,000 years ago. Mark Urban speaks with Miriam Lorbert about the fundamentals of the control loop.

Dave Bittner: Fortinet and Dragos have released a joint report on a commissioned survey by "Food Processing" magazine that looks at cybersecurity in the food and beverage market. The researchers found that the top priority for 93% of food organizations is cybersecurity, followed closely by machine-connected devices. The report states the general tone of the survey results showed the food and beverage industry is working to improve its digital connectivity. Nearly 90% of respondents said their companies give access to company servers and processes to personal devices and other remote devices. As a result, concern is growing over the possibility of cyberattacks. Currently, the perceived threats are malware and ransomware, and the foreseen repercussions are loss of productivity, loss of revenue, service interruptions and compliance issues. Food safety concerns, the threat to population and public health, are surprisingly far down the list. 

Dave Bittner: A cyberattack against Montenegrin infrastructure, which the government has attributed to Russia, appears to have been both extensive and consequential. Bleeping Computer writes - targets include electricity and water supply systems, transportation services, online portals that citizens use to access various state services and more. Power plants have switched to manual operations, and many government IT services have been taken offline to contain the effects of the attack. The country's minister of public administration was at pains to reassure citizens that their data was safe, stating although certain services are currently temporarily disabled for security reasons, the security of the accounts of citizens and business entities and their data is not in any way endangered. Given the kinetic action on the ground, Russian cyberattacks have recently seemed more aimed at punishing nations sympathetic to Ukraine than directed against Ukrainian networks proper. 

Dave Bittner: The U.S. President's National Security Telecommunications Advisory Committee - that's NSTAC - has recommended that the Cybersecurity and Infrastructure Security Agency - that's CISA - require all federal civilian agencies inventory all of their OT assets, MeriTalk reports. The committee stated CISA should issue a binding operational directive similar to what Section 1505 of the fiscal year 2022 National Defense Authorization Act requires for the DOD. That requires executive civilian branch departments and agencies to maintain a real-time continuous inventory of all OT devices, software, systems and assets within their area of responsibility, including an understanding of any interconnectivity to other systems. An up-to-date inventory should be required as part of each department or agency's annual budget process. Once federal agencies clearly understand the vast and interconnected nature of their OT devices and infrastructure, they can then make risk-informed decisions about how to prioritize their cybersecurity budgets to best protect the most consequential of those assets. The White House should mandate periodic reports from CISA on department and agency implementation of this BOD to ensure progress is made. 

Dave Bittner: Nextgov reports that CISA is turning its attention to cybersecurity in the chemical sector with a 100-day sprint focused on improving the sector's resilience. CISA Director Jen Easterly said at a conference last week that she was already impressed with the chemical sector's approach to this issue. Easterly said, it was really telling to me that even back in 2009 how robust the standards were laid out for both physical security but also cybersecurity. It was before cyber was really a thing that this community really understood the importance of a collective approach. 

Dave Bittner: A kinetic incident at a Ukrainian nuclear power plant offers a reminder of the risks of potential cyberattacks. Reuters reports that fires caused by shelling near the nuclear power plant cut power lines to the reactor complex, although backup generators prevented a disaster. Ukrainian President Zelenskyy stated, if our station staff had not reacted after the blackout, then we would have already been forced to overcome the consequences of a radiation accident. Russia has put Ukraine and all Europeans in a situation one step away from a radiation disaster. The situation grew worse over the weekend when heavy Russian shelling cut the nuclear plant off from the Ukrainian power grid. The installation's sole remaining operating reactor is now powering the cooling systems of all the reactors. Should it fail, cooling systems will become reliant on backup diesel generators. The IAEA inspection team is on the premises and has said that power lines should be able to reconnect once fires caused by the shelling are extinguished, the Telegraph reports. While this incident was caused by kinetic activity, it highlights the potential consequences of cyberattacks that target power grids. If disruption of power distribution can occur kinetically, it's good to remember that it can and indeed has occurred through cyberattack. 

Dave Bittner: And since we last spoke, CISA has released a large number of ICS vulnerability advisories over the past weeks. They can be found at cisa.gov/ics under ICS advisories. The latest advisories include products from FATEK Automation, Hitachi Energy, Fuji Electric, Honeywell, Omron, PTC Kepware, Sensormatic, Mitsubishi Electric, Contec and Delta Electronics. Operators should check their systems and review the vendor updates. That's cisa.gov/ics. 

Dave Bittner: I recently had the pleasure of speaking with Dean Parsons from the SANS Technology Institute. We discussed cyberattacks against critical infrastructure, the future of geopolitical conflict, active ICS defense using ICS threat hunting techniques, and the difference between IT and OT incident response. Here's my conversation with Dean Parsons. 

Dave Bittner: So let's start off just by quickly getting a little bit of background on you. I mean, if you're out and about at a cocktail party or something and somebody says, hey, nice to meet you, what do you do for a living - how do you explain that? 

Dean Parsons: Yeah, I guess it starts off by me saying I do cybersecurity for critical infrastructure. And then the next question is, well, typically, what's critical infrastructure? Can you define that? And then it revolves around me pointing at objects, like, see that water tap over there or, you know, the light system outside in the - on the street, the traffic lights? And then we talk about light and heat and power. So yeah, it usually kind of revolves around critical infrastructure and how I work to try to protect that from a cybersecurity perspective. 

Dave Bittner: Do you have the sense that the general public has a greater awareness of these things than they did in the past? I mean, when you think about something like Colonial Pipeline, in my mind, that sort of shone a light on some of the things that we do. 

Dean Parsons: Absolutely has. There's no question that the ICS community is getting more visibility with regards to what attacks are out there or what the impacts could be to, you know, a non-security person, you know, walking around on a daily basis who relies on critical infrastructure daily. So the awareness is definitely out there. I think what we're seeing more so in media is that it's the big bad thing that's going to happen or is happening. And for sure there's active attack and targeting happening on critical infrastructure worldwide, you know - so here in the U.S. and Canada, et cetera, and other parts of the world, as we're seeing with the geopolitical tensions as well. The good news here is that this is defensible, right? These infrastructures are defensible. We can defend these things. And it's not - I mean, it's challenging, yes. But it's definitely doable. 

Dave Bittner: For the professionals, I mean, how much of a responsibility is there to help calibrate the proper messaging of that? You know, to your point, I think it's easy for news and the media to put out the specter that the lights are going to go off or you're not going to be able to get gas or water or any of those types of things. 

Dean Parsons: Yeah. So a lot of even security teams that I have helped built or I work with on a regular basis, they're inundated with, you know, news headlines. And CEOs, you know, rightfully so, are concerned, so they ask questions when they see those headlines. So I think that the real situation comes from threat intelligence. It comes from not necessarily the headlines, but organizations and firms and governments that say, here's the actual attack that we've seen. Here's the methodology behind the attacker that we've noticed and observed. And here's the mitigations to prevent, again, this type of attack we've seen or this type of technique. So there's a difference. It's doable. And threat intelligence is really the best way to understand what's actually happening. And that way, looking and ingesting threat intelligence, it lessens the burden on the noise to security teams, which are, you know, a resourced strain today, so they can get at what's really important immediately rather than sifting through the noise. 

Dave Bittner: Let's dig into some of your specific areas of expertise. I mean, when we're talking about the ICS environment and threat hunting specifically, what's involved there? 

Dean Parsons: Yeah, so - fantastic topic. And I think we're at a situation now in the community where we've passed the point to be mostly just detection-oriented. If we're detecting an adversary in the networks, in the control system, like actually in the control area beyond IT systems, then it - we have an opportunity - right? - to detect and prevent and then respond. But it's more so about proactive defense today, which is threat hunting. We need to assume that there will be a breach at some point. So how do we lessen the ability for the attacker to have an impact in the environment? 

Dean Parsons: Another way threat hunting can be said is proactive defense, where we make the environment less habitable by the adversary if and when they get there. And so threat hunting is really thinking about how an adversary can take advantage and get in your environment using threat intelligence - what we've seen the adversary do already and maybe making some hypothesis based on where they could go in the future. I think that threat hunting is a critical part of our defense moving forward, but it's not a great place to start. 

Dave Bittner: Well, for that organization who wants to spin up a threat hunting function within the organization, what's your advice? Where should they get started? 

Dean Parsons: Yeah, I mean, this is - we've heard this before, but essentially, you're building a house. Build the basement strong, build the walls strong. And build in that order would help. So before threat hunting is established and conducted on a regular cadence, things like industrial control system network architecture, which is absolutely affordable, doesn't require a lot of technology and millions of dollars. Build that basement - strong foundation - and on top of architecture, on top of regular passive defenses for control systems. 

Dean Parsons: And then when you start doing, oh, I have some people, human defenders now looking with visibility in the environment, that's a great place to start your hunting kind of efforts - having those things in place first. Otherwise, you're doing hunting, and you'll get some value from it. There are benefits with hunting right out of the gate. But when you're hunting, you're looking for data sources. And if you don't know what your assets are or what your data sources are or how an adversary could move - again, going back to threat intelligence - then it's not nearly as effective as having those foundational things in place first - your architecture, passive defense, and then human defenders is where you need to go next. 

Dave Bittner: We hear this notion of, within cybersecurity, this idea of living off the land. How does that apply in the ICS arena? 

Dean Parsons: Yeah, so that is - we're seeing that more so now than ever. I mean, we've seen it in 2014 with Havocs, etc., but, you know, in 2022, we're seeing Pipeline, we're seeing CrashOverride in 2016. So living off the land is abusing native controls, native systems, native commands already in the industrial control systems and environments that are used for the purpose of good - for controlling the system, monitoring the system, adjusting the system, adjusting the safety. And so living off the land is taking advantage of those already pre-positioned good things in the environment. 

Dean Parsons: So if we're to detect things like pipeline which - you know, it lives off the land significantly - if we're to detect these kind of things, this is where we have to have active defense. So architecture absolutely needed, for sure. Passive defenses, yes, specific to ICS. But now we're talking about Pipeline and other scalable types of a targeted attacks for control systems that live off the land, and the only way to detect it, respond to it or reduce the habitable nature of the environment is to have active defense, which is human defenders who not only know the threat intelligence side - what the attackers can do when they live off the land - but knowing the industrial environment as well, so knowing a thing about how the control systems work, how a PLC works, the communication path to and from an HMI to the PLC. All of the normal behavior needs to be understood by those human ICS security defenders. 

Dean Parsons: So back to living off the land. It's happening way more now than ever. I think if there was one piece of framework, I guess, that most organizations should understand today is how Pipeline works. We do expect more modules to be built into this adversary framework to not only target industrial sites in the lecture sector, but others. We should assume that will come. 

Dave Bittner: Does the fact that we're seeing that sort of living off the land, does that reflect a maturation of the enemies that there are - is their sophistication, their knowledge of the internal workings growing? 

Dean Parsons: Yeah. So, I mean - so I teach ICS515, and part of that course is, you know, there's malware that exists to target control systems. Living off the land is not using malware specifically. And so yes, we are seeing an educated level by the adversary to target control systems and purely live off the land. And that does a number of things for them. It allows them to specifically go undetected if we're not looking in the network with visibility. And it allows them to bring in less tools to have an impact which they may want to have, you know, to exhibit. 

Dave Bittner: Let's switch gears a little bit and talk about incident response. I mean, I think most people are familiar with that on the IT side of things. Are there specific differences when we're talking about ICS? 

Dean Parsons: Absolutely. So we're all used to the five, six steps of incident response, identification and so on, and all the way up through. There is a definite distinction between IT incident response and ICS incident response. And I would say most of the difference is around things like the containment and eradication. So to be clear, in every step of incident response for control systems, safety is number one. So when you identify things, safety is a result of that and understanding what you've identified and how what you've identified can impact the control system specifically. Containment is really different because containing malware in IT is usually - and rightfully so when we've been doing it well in - for decades in IT - is we've identified a threat. Take that system offline. That system is now not on the network. So whatever that system was doing is now not available. And it works in IT where you take it off the network. You may get forensic data from it, but that system is then wiped, patched, redeployed and possibly the same day, same couple hours, and that's fine. In control systems, identification is critical. 

Dean Parsons: So when we understand what we're dealing with, the containment phase may be, OK, well, maybe we can reduce the ability for that malware or threat on the system to impact other systems. But we may still need to keep that system running to preserve safety and reliability of operations, which means NICS containment may mean, well, there's an IP address going to a C2. We're going to block that on the firewall so it doesn't go anywhere. So the adversary can't get in. Nothing can get out. But the malware can still remain on the system in a contained state, and the eradication piece might not be, take it off the wire right now. Take it off the network. It might mean we have to wait two weeks, a month until an engineering maintenance window comes about where we can actually take that system down safely because we're still using part of that system while there's a threat that's contained. So those are the main things, I think, for sure. And again, safety is considered at every step of incident response in control systems. 

Dave Bittner: Yeah. It's like that old joke about changing the oil while the engine is running, right? (Laughter). 

Dean Parsons: Yeah. Absolutely. And the - even the responders, the people responding to control system events, are - there's more people and more teams. So when there is a physical possible ramification of an event - so a safety instrumented system is on to keep people and infrastructure safe physically from blowing up and causing mass damage - every safety instrumented system is involved in an incident or could be impacted. Now you have not only cyber defenders responding and understanding what's happening, you have physical safety teams on site, emergency incident response from a physical perspective on site that are responding and need be aware and have an understanding of what the capabilities are. So incident response in control systems are far beyond what's expected in IT, more people involved. I think the way to get there, find the nuances as you determine what's different in between both environments, is the tabletops that can be conducted in control systems - a lot of value there with those cyber folks and the engineers and the VP of operations and stakeholders and owners of the plant and the safety teams as well. 

Dave Bittner: If I'm responsible for, you know, keeping the SOC running in my ICS environment, how do I take care of the care and feeding of both of the IT side and the OT side of the house? 

Dean Parsons: There shouldn't be two sides - right? - the IT side and the ICS side... 

Dave Bittner: Yeah. 

Dean Parsons: ...If an organization which has critical infrastructure is in business or an OT environment's in business and that is the business. So collaboration is critical for this main reason. The adversary we've seen, not all the time but frequently, chooses to enter the IT network first. They target the IT network, compromise IT first and then work their way somehow into the control system. Because of that, you know, the person operating the SOC for IT and ICS, it's critical or helpful and beneficial if that person and that team has the visibility from IT and the ICS. So technically, they need - they should have or could have and be beneficial if they had the network events from IT and the network events from the ICS to get the full visibility of the network. And I refer to this as the ICS cyber kill chain. Not always, but it follows the flow of, you know, the IT kill chain expanded upon where the IT network is traditionally seen as the first point of entry. And if IT's only looking at things that look like IT and not potentially ICS, then when an adversary gets to the control network, then the ICS defenders, if you will - you know, they don't have full visibility, have 50% of the visibility, and they've lost time because the adversary is now halfway in - right? - at the front door. 

Dave Bittner: So before we wrap up, can you give us a little preview of the course that you have put together? There's a new SANS course, which is ICS security essentials for managers. Can you give us a little preview of what that's about? 

Dean Parsons: Absolutely. So the ICS418, security essentials for ICS managers, is really, we're inviting people that come into ICS from IT. We need those folks that are in the engineering kind of realm to step up to responsibilities for ICS and the in-place kind of managers already doing ICS security. This course is really for those individuals that manage cyber risk - the management level. So in that course, we talk about how to build a team, how to expect and what to expect from a technical team who may have done ICS515 as an example. And we walk them through how to present to the board, understand what the threats are today. And it's a course that's also management level, but it's hands-on. So we have work exercises throughout that you walk away from this two-day course with, here's the things that may be gaps in my environment today. And we allow in class and the exercises for them to - for the students to fill some of this information out with the things they already know about their environment. So you walk away with, you know, back to the office Monday morning, hitting the ground running with here's a prioritization of what I should look at based on the threat landscape out there from a management perspective. 

Dave Bittner: All right. Well, best wishes on the new course. Dean Parsons, thanks so much for joining us. 

Dean Parsons: Absolutely. Thank you so much for having me, everybody. 

Dave Bittner: Our thanks to Dean Parsons from the SANS Institute for joining us. 

Dave Bittner: Fun fact, the first industrial control system was more than 2,000 years ago, and it involved a toilet. For all the details, here's this week's Learning Lab, with Mark Urban speaking with Miriam Lorbert about the fundamentals of the control loop. 

Mark Urban: Thanks, and hello again. In today's episode, we're going to talk some more about the core concepts in industrial control systems, still building on those basics. And I'm joined today by Miriam Lorbert, a principal industrial consultant here at Dragos. Prior to Dragos, Miriam spent some time working in General Electric as a cybersecurity engineer. She served as a process control engineer in the energy sector, and she's been a part-time adjunct instructor at John Hopkins Whiting School of Engineering. Welcome, Miriam. 

Miriam Lorbert: Thank you so much for having me. 

Mark Urban: Yeah, our pleasure. Miriam, besides consulting with customers on security architectures, cybersecurity frameworks and other topics, you help develop courses for Dragos Academy. What is Dragos Academy? 

Miriam Lorbert: Dragos Academy is part of the Dragos offering, which is free to platform users to get educated on industrial control systems and their environments, and how we can secure those systems and architectures. A lot of what we're talking about today is a summary of some of those topics in our ICS OT training. 

Mark Urban: Got you. And we were looking through some of the training and found material that I thought would be great for this education - for this audience here on the "Control Loop" podcast. And maybe that's a good place to start with - control loop. And a control loop is a fundamental concept in industrial systems. Could you help explain it to us? 

Miriam Lorbert: Yeah. And definitely a necessary concept for this podcast, given the name - a control loop usually applies logic to how a process is controlled. So in an industrial setting, it usually applies some sort of predefined physical reaction to real-time changing environment. There are a few main components required that make up a control loop. First, we have a sensor, which is monitoring a particular variable, kind of like our home thermostats monitor the temperature. Second, we have an actuator, and that causes some sort of physical reaction to change the conditions based on a predefined, set point. So, for example, we would love to maintain a particular temperature in our homes, and this is how we would go about that. As control loops' complexities grow, transmitters and controllers have been added. So a sensor would measure some sort of process. A transmitter would transmit that measurement to a controller. Often, controller encompasses either a programmable logic controller, also known as a PLC, or a distributed control system, also known as a DCS. That controller is programmed with specific logic to decide how to communicate to the actuator to change or not to change the process. So in summary, a control loop is a discrete operation of constantly monitoring and managing specific variables within an environment to control a process. And zooming out, industrial systems are a series - vast amalgamations of control loops that manage very focused and discrete operations. 

Mark Urban: Okay, gotcha. So sensors and actuators at its core and then getting a little bit more complicated as you move into, you know, different industrial environments - can you give us a couple examples of control loops in the live world? 

Miriam Lorbert: Yeah, let's take natural gas as an example. So we use it to fuel furnaces, power stoves at home, light up our grills, and power plants use it to generate electricity. Natural gas starts in the ground. There are a lot of different ways to drill and extract it. Once extracted, it's collected in low pressure pipes that start to aggregate into larger pipes like streams into a river. And the movement of the gas and the pipes need to be monitored. So there are sensors that measure the pressure and transmitters that send the data to controllers. And the controllers open and close valves based on the target or what's known as a set point. That's just one set of control loops. Then we can look at the processing plant. So when the natural gas gets to its destination for processing, it goes through many distinct processes to purify the gas. Each process is one or many different control loops that focus on a particular operation. Some examples would be removing oil and condensate from the natural gas or removing water or separating liquids from the gas, removing sulfur and/or carbon dioxide. Each of those processes involves several distinct control loops. 

Miriam Lorbert: Those control loops take the science of purifying natural gas and apply them at an industrial scale to create a quality natural gas end product. More control loops will monitor and manage the flow of the gas downstream to its eventual destination in your gas stovetop or furnace. And it will monitor pressure, open and close valves, loop after loop after loop - science automated at scale. 

Mark Urban: OK, science automated, OK. And, you know, so that's, you know, applied at a very high scale. How do control loops come about, you know? Can you give us some background there? 

Miriam Lorbert: Yeah. How far back would you like to go? 

Mark Urban: How far back can you go? 

Miriam Lorbert: Well, about 250 BC, a Greek inventor named Ctesibius was challenged with creating a clock that was capable of running autonomously for 365 days a year while maintaining accurate time. So he built a toilet. 

Mark Urban: What? Excuse me - a toilet to tell time? 

Miriam Lorbert: No, not exactly. He built a water regulator - a float - to monitor the level of water and to control the opening and closing of valves to maintain constant water pressure. A lot of similarities to a modern toilet float that monitors the level of water and opens and closes the water valve. The consistent flow of water was what he used ultimately to measure time. 

Mark Urban: So a toilet-based clock from 2,200 years ago was the first known example of a controlled industrial system. 

Miriam Lorbert: Yeah, pretty much. 

Mark Urban: OK. I'll take your word for that. Let's fast forward a bit. When did control loops become more computerized - less reliant on toilet technology? 

Miriam Lorbert: Well, as you might imagine, automobiles had a lot to do with it. So General Motors - GM - was looking to automate its auto factories. And so it issued a challenge. It was seeking a solid state system that was flexible like a computer, but priced competitively, with a relay logic system that could control specific processes. It had to be easily maintained and programmed in line with the already accepted relay ladder logic way of doing things. It had to work in industrial environment with all the dirt, moisture, electromagnetism and vibrations, etc., and it had to be modular in form to allow for easy exchange of components and expandability. 

Mark Urban: All right, so they issued a challenge, like a contest. 

Miriam Lorbert: Yep. And as a result, the first programmable logic controller, or PLC, was created in 1968 by a company known as Modicon - the first modern control system that brought us to where we are today. 

Mark Urban: OK. Where are we today? 

Miriam Lorbert: It's truly awe-inspiring to see how much we've grown in the realm of control loops and controllers because we're able to take these massive processes, like the natural gas pipeline grid throughout America or electrical power grids that support millions and millions of customers, and segment them into distinct processes and controllers. Some entities even run completely autonomously. 

Miriam Lorbert: While the applications of control loops have grown immensely since more than 2,000 years ago, we do need to mention the caveat that control loops and controllers are typically ran on legacy equipment and architectures that have since been retrofitted for remote connectivity or with external connectivity. And that's where all of us at Dragos come in - to help circulate foundational information, just as we have discussed some in this podcast, and raise awareness of vulnerabilities where we can. 

Mark Urban: Well, Miriam, thank you. That's an interesting not only kind of how they work, but the background and history is pretty interesting - you know, going back to Greek philosophers and as well as talking about as much as is expanded, you know, that connectivity can be a challenge, and understanding is the first concept of being able to change. So thanks for that information. Appreciate your time this morning. 

Miriam Lorbert: Yeah, thank you very much. 

Mark Urban: All right, that was Miriam Lorbert with "Control Loop." I'm Mark Urban. 

Dave Bittner: And that's "Control Loop," brought to you by the CyberWire and powered by Dragos. For links to all of today's stories, check out our show notes at thecyberwire.com. 

Dave Bittner: Sound design for the show is done by Elliott Peltzman with mixing by Tre Hester. Our senior producer is Jennifer Eiben. Our Dragos producers are Joanne Rasch and Mark Urban. Our executive editor is Peter Kilpe. And I'm Dave Bittner. Thanks for listening. We'll see you back here next time.