Control Loop: The OT Cybersecurity Podcast 6.15.22
Ep 2 | 6.15.22

ICS training and education is a maturing domain.


Dave Bittner: It's June 15, 2022, and you are listening to "Control Loop." In today's OT cybersecurity briefing, The U.K. attorney general discusses hacking back in defense of critical infrastructure. Ethiopia says it stopped cyberattacks on its Nile dam. Recommended cybersecurity improvements for dams in the southeastern U.S. CISA and its partners issue guidelines for evaluating 5G implementation. A look at water systems security. Deloitte opens a smart factory at Wichita State University. Tim Conway from SANS discusses workforce development for OT cybersecurity personnel. New developments in cybersecurity, professional education and the skills shortage. In the Learning Lab, Mark Urban is joined by Nick Shaw to give an intro to OT.

Dave Bittner: U.K. Attorney General Suella Braverman declared that the country reserved the right to conduct cyber counterattacks when key services are targeted by nation-state adversaries, thereby officially extending the right of retaliation under international law to the digital world. It's worth noting that in military doctrine, a counterattack is a defensive operation. Traditionally, there has been an unofficial global agreement that military force will not be used to defend against digital attacks or espionage, especially since attribution is difficult. However, recent attacks on critical infrastructure and government networks have put that unspoken understanding to the test. In her remarks, Braverman cited recent incidents like the July 2021 Microsoft Exchange breaches and the SolarWinds attack. And she named four sectors as particularly vulnerable - the energy sector, essential medical care, the supply chain and democratic processes. CPO Magazine notes that while none of these areas have been directly targeted by nation-states, aside from Russia's attacks on Ukraine, they have increasingly fallen victim to ransomware operations. And North Korean advanced threat groups have been observed using such attacks to fund the government. 

Dave Bittner: An audit by the Tennessee Valley Authority Inspector General concluded that its flood control and water management dams were at risk from cyber-related and physical vulnerabilities, FCW reports. The federally-owned electric utility corporation stated in its report, in summary, we found, one, no clear ownership of the non-power dam control system, two, vulnerable versions of operating systems and control system software, three, inappropriate logical and physical access, and four, internal information technology controls were not operating effectively or had not been designed and implemented. An example of a cyberattack against a non-hydroelectric dam occurred in 2013 when the small Bowman street dam in Rye, N.Y., was targeted by hackers. The U.S. Justice Department later attributed this incident to members of the Iranian government's Islamic Revolutionary Guard Corps. The hackers in that case weren't able to manipulate the dam itself, and even if they had opened the sluice gate, it likely wouldn't have caused any significant damage. Even so, the incident demonstrated an interest from state-sponsored actors in targeting this type of infrastructure. 

Dave Bittner: The official response to the Bowman street dam incident offers some interesting contrasts with current policy. It was discussed at some length, for example, at the Georgetown Law Center's Cybersecurity Law Institute in May of 2016. A panel that took up cyber threats to infrastructure said the Bowman street dam essentially stops a brook from babbling. It's small and basically designed to keep some basements from flooding and perhaps an adjacent little league field playable. But the incident shows the opportunism of nation-states. Everything can be a target. The attack on the flood control dam was attributed to Iranian actors. While they were intruding into the dam's control system, however, a more serious attack was underway. A panelist said, while the Bowman Street Dam was being hacked, so was the financial services industry. And they got no help from the government. 

Dave Bittner: The complaint was that the authorities told infrastructure operators to report incidents, but their assistance seldom went beyond that. How many times, a panelist asked, are you going to let someone rattle the handle? While there's bound to be room for improvement, public-private cooperation for better infrastructure cybersecurity has advanced considerably since 2016. Today, it's unlikely in the extreme that the authorities would take no action on reports of cyberattacks against critical infrastructure. 

Dave Bittner: Hydropower dams have their own cybersecurity concerns. In another part of the world, Ethiopia recently said it stopped cyberattacks on its Nile dam and some financial institutions, the Addis Standard reports. AI Monitor says that Egypt's government has not officially responded to Ethiopian accusations that it's behind any such cyberattacks. The Grand Ethiopian Renaissance Dam and the Nile water rights it affects have been a point of contention between the two countries. 

Dave Bittner: U.S. Cyberspace Solarium Commission member, Representative Jim Langevin, Democrat of Rhode Island, said at a Foundation for Defense of Democracies event last week that the Environmental Protection Agency was unprepared to protect water infrastructure from cyberattacks, CyberScoop reports. Langevin said, knowing what we know about the cyberthreats facing the water sector, this status quo simply cannot continue. Mark Montgomery, former executive director of the Cyberspace Solarium Commission and current senior director of the Center on Cyber and Technology Innovation at FDD, added, our vision here is a sector-led organization, and that's because at this time, without getting into all the nitty-gritty, the EPA's water cybersecurity team is - you can count the number of people on one hand. Three-Finger Brown might have been able to cut them all in one hand. The idea is that it is because of that lack of capacity right now in EPA that we need a sector-led organization to manage the development of mandatory cybersecurity standards and oversee compliance with them. But because you have to have federal oversight, this approach does account for federal oversight by that limited EPA team, which, separately, we recommend growing significantly. 

Dave Bittner: An EPA spokesperson said in a statement, the EPA is committed to using its available authorities and resources to strengthen cybersecurity across the water sector. Recent events have highlighted the importance of this effort, and the agency is taking a multipronged approach in close partnership and coordination across the federal government and in collaboration with state agencies. 

Dave Bittner: MITRE has released its System of Trust framework to protect supply chains. The organization stated in a press release, the SoT framework builds a basis of trust by identifying the three main trust aspects of supply chain security - suppliers, supplies and services - then identifying and addressing the 14 top-level decision risk areas under them associated with trust that agencies and enterprises must evaluate and make choices about during the full life cycle of their acquisition activities. Leveraging the full breadth and depth of our expertise, industry efforts and government research, the SoT framework drills down into these 14 top-level risk areas and investigates as many as 200 risk sub areas by addressing over 2,200 detailed questions. 

Dave Bittner: NSA and others have pointed out that 5G technology isn't simply the latest IT standard for smartphones. 5G will find its uses wherever devices are connected, as they are in OT networks. CISA and its partners in the Department of Homeland Security's Science and Technology Directorate and the Office of the Undersecretary of Defense for Research and Engineering have released Version 1 of the 5G Security Evaluation Process Investigation. It outlines a five-step process organizations should follow as they implement 5G. The report says step one calls for a use-case definition to identify 5G subsystems that are part of the system - component configurations, applications and interfaces involved in the operation of the system. In step two, agencies should define the boundary to identify the technologies and systems requiring assessment and authorization, taking into consideration the ownership and deployment of the products and services that comprise the use case. 

Dave Bittner: The third step, after determining the scope of the assessment, is to perform a threat analysis of each 5G subsystem with a view to mitigating the risks associated with it. At step four, an agency should consult relevant federal security guidelines and create a catalog of that guidance. And finally, in the fifth step, the agency applies the guidelines, identifies any gaps in security guidance for ways to address them. It seems a common-sense approach with an appropriately bureaucratic bent. But CISA hopes that it will provide an approach that's both uniform and flexible. CISA invites feedback, and the deadline for comment is June 27. 

Dave Bittner: The Record by Recorded Future reports that tech manufacturing giant Foxconn said its factory in Mexico is slowly returning to normal after a ransomware attack crippled the facility in May. The LockBit ransomware group claimed to have attacked the company's offices in Tijuana last month. They threatened to leak the data stolen during the attack by June 11. The LockBit ransomware group is known as one of the most active when it comes to compromising organizations with an industrial control system, operational technology environment, BleepingComputer reports. A spokesperson from the Taiwanese company confirmed the attack. 

Dave Bittner: Deloitte has opened a Smart Factory in Wichita, Kansas, designed to advance the future of manufacturing by studying and demonstrating manufacturing techniques in a variety of applications on a shop floor. Deloitte said in a press release, organizations that engineer an end-to-end smart manufacturing operation can increase efficiency, sustainability and cybersecurity, build resilience and create new levels of growth and competitive advantage. The founders of the project include AWS, Dragos, Infor, SAP, Siemens and Wichita State University, along with participants Check Point, HPE, Tenable, ServiceNow, UiPath, Verizon and others. 

Dave Bittner: Tim Conway is curriculum director for industrial control systems at the SANS Institute. I caught up with him at the SANS ICS Security Summit in Orlando, Florida. 

Tim Conway: Yeah. So this is an area when - in electric utilities, where I worked for a number of years, we were a combined gas interstate pipeline and distribution and a electric transmission generation and control center ops. So when we would look at cybersecurity training, in many cases it was - you would buy a system from a vendor, and that vendor is somebody that - it's not like buying your home computer, where you're trying out an Apple one day, and then, you know, a year from now you don't like it, so you buy a Dell or an HP. You can think of that like you're dating, those different technologies. 

Dave Bittner: Right. 

Tim Conway: In critical infrastructure, you're marrying a solution provider. You're working for years to build and architect and design and engineer a Honeywell system, a Johnson Controls system, a Westinghouse, a GE system, and that is what you're going to be on forever. So they are like a partner in everything with you, and you look to them for training on how do we use the system? How do we design it? How do we support it? And then discussions on, how do we secure it? And across the vendor space and across the asset-owner space, we all sort of walked into security at the same time of - we should be doing this different. We should be doing it better. We should be requiring authentication and requiring area of control responsibilities and very specific, unique items. 

Tim Conway: So we were sort of learning as we went. And we would take things that were like what we were doing. We would take IT security courses 'cause they were available, even though that's not what we were doing. And then we would go to a class during the day, and we would meet with our teams at night and say, this doesn't apply here. This doesn't apply here. You can't use these tools in this environment, or it doesn't work that way in this type of critical infrastructure. Industry leaders like Alan Paller and Mike Assante came together and said, well, how do we stop that? How do we train the way we should be training for our systems? And it's really what started this event that we're at now, the Industrial Control System Summit, where we started taking practitioners in the space who are doing it and, really, case studies and lessons learned of how are you doing this so you can present it and we can all learn from it and try to do something similar? Those were the courses back then, the presentations and learning from your peers. 

Tim Conway: After we ran summits for probably seven years, then there became a discussion of - this community is growing, and the need is growing, and the impacts to a critical infrastructure attack are becoming apparent. We should really have dedicated courses and a curriculum. And again - same people - Alan and Mike started this process of, let's have an industrial control system-specific course, and let's focus on from ground up foundational IT and OT, and that was our first class. And one of the things that Mike really, really kind of took hold of was - it shouldn't just be a class. This is a - you know, we're working with professional engineers. We're working with critical infrastructure, national assets and national interest and strategy. We should probably have a certification that somebody could look to and say, do you have the minimum requirements to come and work on this type of system? And let's start with discussions like, do you understand the safety risks to yourself? Do you understand the safety risks that you could pose to others and the communities that we serve? Let's make sure you understand those things, and then let's start talking about - do you have the ability to come and work in safe - safely work in these environments and kind of do cybersecurity work? So you're not just going to come in like it's an IT system and start doing patches and then find out you tripped a unit offline... 

Dave Bittner: Right. 

Tim Conway: ...Or you caused a problem or a fault. 

Dave Bittner: Right. 

Tim Conway: So just really focusing on the right types of learning objectives that are unique to industrial control systems. And it's sort of grown from there. We were going to make one course and a certification that we could validate some people and say, these people hold credentials that now we can allow them to come in. And then we started seeing a need for another course as they started architecting differently and putting in sensors and sort of looking for evil and starting to see odd traffic and saying, how do we know what to look for, and what do we do using traditional intrusion detection systems? They're not catching operational data or variances or deviations. Using traditional and intrusion prevention systems - we would never do because it could - or potentially impact the process. So what should we be doing? And people like Robert M. Lee kind of said, I've worked in this space of seeing data and looking at data and understanding processes. And I've worked in this intelligence space where we're building actionable playbooks. And we should probably have a class that addresses that. 

Tim Conway: So we started the class after class after class, grow this curriculum. And now we've started to create this community of, well, what do you mean you want to be an industrial control system cybersecurity professional? Do you want to be on the protocol side, on the network monitoring and the OT visibility? Do you want to be in instrumentation and control and kind of designing and building things in different ways and more cyber secure? Are you a generalist, where you're going as a system integrator to a number of different sectors where you're going to know a little bit about all of them to do work safely? Well, maybe you need this kind of general ICS-410 course. And we've started to see this community grow and populate that way, really starting from this event a number of years ago with Mike and Alan. It's been quite a journey to see. I know the way we used to consume training and prepare our teams for the battle - vastly different just in one generation of - I've worked on the space for about 23 years now, and it is significantly different from when I came in first. 

Dave Bittner: As you look towards the future, what are some of the things that you think of as being gaps that need to be filled? 

Tim Conway: Yeah, you know, so I think in kind of a - this is something just like life that goes through different cycles and waves. You know, when you're a teenager, what life looks like to you when you're single, out of college or in college, when you're married, when you have kids - the different stages of life that you go through - there's - certainly, as a industrial control system cybersecurity community, we have in times of peace, where there's been three, four, five, six years of no specific industrial control system malware - there's no specific attacks that are happening. And we get to focus on longer-term, bigger problems. How do we build regulation to make sure that everybody's coming up to a common bar? And we know that regulation is going to take 2 to 3 years to get approved and 2 to 3 years to go in and actually get implemented. And then it'll take some years for people to go out and look at it, assess it and determine whether you did what you were supposed to do. So that's a 10-year road before you're starting to get to where you wanted to be. And where you wanted to be was 10 years ago. 

Tim Conway: So where you need to be by then is different. And that is a long-term game that you're just going to continue to chase for every single sector, whether it's nuclear, electric, natural gas, where you've seen regulatory environments go. Those are things that we get to focus on and talk about when everything is good. When things are happening now, where we are seeing industrial control system-specific malware coming out that is being used in active exploits, a couple a month, that is something that this community is not used to. And when they're coming out at that pace and you say, well, here's how you fix it - but you can't just drive around all your facilities and all your infrastructure and start installing patches and start closing off communication paths because inherently the way we operate them, you would potentially cause outages by behaving that way. So you have to respond in very specific ways. And you have to architect solutions and methods of operating through an attack so that you can successfully deliver your service while an attack is occurring. You need to understand how you can operate when your infrastructure is being compromised and misused or misoperated. So as you look to Ukraine now and how that country is continuing to operate, provide service and provide electric service for the customers that are there, for the people who need it to sustain life while it's still actively being physically attacked, logically attacked - and as other nations and countries, we can look to that, and we can start learning from it and saying, we had these big-term plans, you know, these 20 controls we were going to try to implement over the next 10 years. But because of the state of the world right now, what are the four or five things that we should be doing immediately? We need to move from long-term strategic in time of peace to, these are the five things we're going to go focus on right now as a rapid response. 

Tim Conway: Our nation, other surrounding nation, ally countries that are helping and supporting Ukraine that is being targeted, the unimaginable things that are happening there - that is informing and influencing what actions we are focusing on now. And that will include what technologies we focus on and reprioritize those, where investments and corporations - where they may have before been highly investing in IT because of constant kind of - that's where the threat from spear-phishing and data loss and intellectual property theft and potential brand damage. And then the new shift and focus on ransomware and what - how - what's the risk of a ransomware attack and what we have to pay? And now with the focus of what's happening in Eastern Europe, it's different conversations. It's - those IT systems are how you manage your business. The OT systems are why you're a business. So the types of technologies you're using to move molecules or move megawatts or the things that you're delivering - these OT systems are starting to get a lot more attention because it's why these businesses exist in critical infrastructure and key resource. And just the shifting sands back and forth of what things look like for this community when we're in a time of peace and nothing's happening versus the - there are things that are being seen now, and it's at a much, much more rapid pace of different actions, different strategies that need to exist. 

Dave Bittner: All right. Well, Tim Conway, thank you so much for taking the time for us today. 

Dave Bittner: In today's "Control Loop" Learning Lab, Mark Urban checks in with Nick Shaw for a refresher on the OT basics. 

Mark Urban: Thanks, Dave. I'm here with Nick Shaw from Dragos. Nick, can you introduce yourself, please? 

Nick Shaw: Yeah. Hey, everyone. My name is Nick Shaw. I'm an advisory solutions architect here at Dragos. So I work with customers across North America on their journey for improving their OT cybersecurity. 

Mark Urban: Excellent. Excellent. Now, Nick, I'm a longtime cybersecurity guy on the IT side of cybersecurity. Joining Dragos has exposed a whole different world of operational technology. Can you give us a basic understanding of what is OT, operational technology, at a simple level? 

Nick Shaw: Yeah, excellent question. So when I look at what IT focuses on, IT focuses on data and system security, whereas OT, or operational technology, adds a physical element to it. So you're looking at physical processes, the machines that are affecting a physical world with automation to carry out certain tasks within a facility, really such as a manufacturing plant or factory. So there's really subsets of OT. You have building automation systems that look at possibly the HVAC of a data center or environmentals of a building where an industrial control system, or ICS, is used to automate a system for making parts of cars, mix raw materials to make cheese and really take that raw material and turn it into a finished product. So ICS is kind of a subset of OT as the entire umbrella where there are other OT processes that are separate from an industrial setting. 

Nick Shaw: And really when we look at industrial control systems, we're looking at things that affect safety, productivity and quality. And it's product safety, people safety, quality of the product we're producing and making sure that from a productivity perspective, the plant is up and running as much as we can. It has minimized unplanned downtime. You know, we're being proactive with how we're planning maintenance and that downtime. 

Nick Shaw: And really when I look at the landscape of OT and ICS, those terms get used interchangeably, but I focus mostly on industrial control systems and the various manufacturing processes or oil and gas and electrical sectors. And I know when we talk about the whole landscape OT, IT and OT kind of get mixed in with where does IoT play in that? And IoT frequently gets confused with ICS and the OT environment that it gets mixed in the same bucket. They're really not the same. So if you're looking at an Alexa, for example, it's not the same as an industrial control system that can affect loss of life or the products that we're consuming that are produced in these environments. 

Mark Urban: So, OK, so OT, operational technology, everything from, you know, controlling processes at a chemical plant or a cheesemaking plant or an oil and gas pipeline or, you know, manufacturing processes, that's kind of operational technology, industrial control system, ICS. And then you're drawing a contrast to very popular term IoT, which, you know, Alexa is a good call on that. And you have two completely different things. Can you talk a little bit about - OK, so if you have the OT world, you have the IT world that we're familiar with, what are some of the differences between, you know, especially from a security standpoint and some of the fundamental technologies, what are some of the differences in those two worlds? 

Nick Shaw: Yeah, absolutely. So IT is more of a productivity inconvenience if you can't access email or Slack for a short period of time. OT systems are purpose built to carry out a task, whether you're assembling parts into a spark plug for an automobile or measuring out raw materials for a batch of your favorite beverages. I've spent a lot of time in manufacturing environments that I've seen a lot of food get manufactured, pharmaceuticals and automotives. So those are really the machines, robots or processes that assemble the raw materials into a final product that hits your shelves eventually and you see in a grocery store or something like a supermarket. So when you're looking at OT environments, there's a lot of legacy systems, things that have been installed in the '80s and '90s using older technologies, unique protocols. And, quite frankly, these systems are insecure by design. These protocols are unauthenticated and unencrypted by nature. 

Nick Shaw: So often when you're looking at security, the adversaries we come across in operational technology environments can really utilize that native functionality of a control system versus relying on exploiting a vulnerability. So when you look at the different controls we have in place for an OT environment, we really want to monitor communications that are occurring and have what's called a defensible architecture for how we want to protect that shop floor environment from remote access, third-party risk, the IT environment and vice versa. So when you look at the different considerations and really where you focus in industrial environments, the first and foremost thing is really the availability of the system. The goal is to really minimize unplanned downtime while producing a high-quality product in a safe and reliable manner that's safe for the product you're consuming, you know, that you're putting out to consumers and also from a people safety that you're not causing unsafe conditions in these environments. 

Nick Shaw: So in a previous role during an unplanned outage, IT actually asked me, like, how many users is this affecting? And that's really not applicable to OT environments. When you look in OT environments, more focused on what is that cost of an unplanned downtime? And if you find the right person in the organization, you can usually get an answer for how much a facility costs if it is down for a period of time. And that downtime can be in the hundreds of thousands of dollars per hour, depending on the products being made in that facility or in that process. So there's considerations, of course, you know, when you look at that unplanned downtime for what do you do with the raw materials? Is it something that is perishable and you need to return to a cooling environment such as refrigeration? And so it's a little different consideration when you're worried about the availability of email or a communication platform versus, like, actual physical processes interacting with the goods that are being made into a final product. 

Mark Urban: So shutting down, you know, an employee's laptop for, you know, 45 minutes for a patch upgrade is much different than changing or updating and patching a component of a highly complex, you know, manufacturing facility, is that what you're saying? 

Nick Shaw: Absolutely. I think if you look at supply chain constraints right now, you look at facilities that are running around the clock 24-by-7 minimal downtime. If they're down for two days out of the year, that's a big deal. You know, when you look at scheduling downtime with that predictive maintenance, depending on how much data is at your fingertips to kind of predict when you're going to need to go down to be proactive so you're not going down for a longer period of time if a piece of equipment were to break, there's definitely a different consideration for sure. 

Mark Urban: So you're talking about industrial systems. Can you give me a couple examples of different types of industrial systems? 

Nick Shaw: Absolutely. And we'll talk with a couple different industry verticals in mind. So I'll start with water. One of the things that was top of mind last year, especially around the Super Bowl, was a water supply and purification system. Oldsmar was in the news last year where an industrial control system was accessed by, you know, an outside party. And the control and measurement of lye for the water supply in the system and the process of purifying water was actually modified for the parts per million that was going to be introduced into that system. Now, if it had gone unmonitored or undiscovered for a period of time, it could have resulted in a possible poisoning for the residents of Oldsmar. But luckily, you know, a human was observing this on an HMI - and we'll get into what an HMI is - was able to catch the issue and modify that back to the correct parts per million. 

Nick Shaw: So in a water supply and purification process, there's different devices that would be controlling and monitoring and observing the process. And it's very much applicable to the same sorts of devices that would be carrying out those tasks in other different verticals, like manufacturing. You have different types of manufacturing processes, whether it's a discrete manufacturing that takes a bill of material of parts and can assemble those parts into a larger part. The spark plug, for example, has a bill of material of parts that comes into a bigger part that gets assembled into an engine. Those things can be taken apart. Those things can be reworked. But when you get into process manufacturing or batch manufacturing, you're taking a lot of raw materials - sugars, proteins, dairy, otherwise - assembling those into a final product. Those things can't really be disassembled in a process. 

Nick Shaw: So if you have something where a set point gets mismanaged, you have to actually scrap the batch instead of reworking that. And that is a loss of materials that you were planning as part of your supply chain. And then for, like, oil and gas, of course, you've got refineries, compressor stations, different main lines and valves for moving around, you know, natural gas and oil and petroleum. And, of course, you've got main control centers really monitoring those remote off sites. You've got offshore platforms, oil wells, drill sites and pump stations on these pipelines. 

Mark Urban: You know, that's kind of eye-opening about how much of these technologies help operate our lives, you know, from water to manufacturing systems, our food, our pharmaceuticals, the electricity that we have, all relying on these different systems. And we're running up on time, and it feels like we've just kind of scratched the surface. So if it makes sense, if we could have you come back to kind of dive into some of these operational technology kind of components and layers and how we can think about it and understand them better. 

Nick Shaw: Absolutely. And we can dive into various device types. I've thrown out a couple there about a PLC and an HMI, and we can talk about what's the function and the intention of those different assets as it pertains to different verticals as well, how they are, you know, aligned within the various enterprise reference models like the Purdue Enterprise Reference Architecture and how we look at securing those assets. 

Mark Urban: Sounds good. Nick Shaw, thanks very much. Dave, back to you. 

Dave Bittner: And that's "Control Loop," brought to you by the CyberWire and powered by Dragos. For links to all of today's stories, check out our show notes at, where you can also subscribe to the monthly "Control Loop" newsletter. Sound design for the show is done by Elliott Peltzman, with mixing by Tre Hester. Our senior producer is Jennifer Eiben. Our Dragos producers are Joanne Rasch and Mark Urban. Our executive editor is Peter Kilpe. And I'm Dave Gardner. Thanks for listening.