Providing a safe and secure OT infrastructure.

Dave Bittner: It's September 21, 2022, and you're listening to "Control Loop." In today's OT cybersecurity briefing, the Palestinian hacktivist group GhostSec compromises Israeli PLCs. North Korea's Lazarus Group targets the energy sector. The White House issues a memorandum on supply chain security. And CISA issues advisories on ICS vulnerabilities. In our featured interview, Rachael Conrad of Rockwell Automation talks about how industrial automation organizations can achieve their connected enterprise by providing a safe and secure infrastructure. And in the Learning Lab, Mark Urban of Dragos, in part one of a two-part segment, explores the scale of the generation of electricity.

Dave Bittner: The Palestinian hacktivist group GhostSec earlier this month claimed to have compromised 51 Berghof PLC devices - that is, programmable logic controllers - in Israel, according to researchers at OTORIO. The researchers found that PLCs were exposed to the internet and had default passwords, writing, although access to the admin panel provides full control over some of the PLCs' functionality, it does not provide direct control over the industrial process. It is possible to affect the process to some extent, but the actual process configuration itself isn't available solely from the admin panel. From our research, we concluded that Berghof uses CODESYS technology as its HMI and is also accessible via the browser at a certain address. From our observations of GhostSec's proofs of breach, we did not know whether GhostSec gained access to the HMI, but we've confirmed that the HMI screen was also publicly available. 

Dave Bittner: While it's not clear if the hackers actually had the ability to manipulate the industrial processes, the group said on Twitter that they decided not to alter pH levels in water in order to avoid harming civilians. The researchers continue - the fact that the HMI probably wasn't accessed nor manipulated by GhostSec and the hackers were not exploiting the Modbus interface shows an unfamiliarity with the OT domain. To the best of our knowledge, GhostSec hadn't brought critical damage to the affected systems, but only sought to draw attention to the hacktivist group and its activities. Despite the low impact of this incident, this is a great example where a cyberattack could have easily been avoided by simple, proper configuration, disabling the public exposure of assets to the internet and maintaining a good password policy - especially changing the default login credentials - would cause the hacktivist breach attempt to fail. 

Dave Bittner: Cisco Talos warns that North Korea's Lazarus Group has been targeting the energy sector in the U.S., Canada and Japan. The threat actor uses the Log4Shell vulnerability to compromise VMware Horizon servers. The researchers stated, the main goal of these attacks was likely to establish long-term access into victim networks to conduct espionage operations in support of North Korean government objectives. This activity aligns with historical Lazarus intrusions targeting critical infrastructure and energy companies to establish long-term access to siphon off proprietary intellectual property. 

Dave Bittner: Talos also describes the threat actor's post-exploitation activities, writing, successful post-exploitation led to the download of their toolkit from web servers. The same initial vector, URL pattern and similar subsequent hands-on-keyboard activity have been described in this report from ON.Lab from earlier this year. There are also overlapping IOCs between the campaign described by ON.Lab and the current campaign, such as the IP address, which was used as a hosting platform for the actor's malicious tools. Although the same tactics have been applied in both attacks, the resulting malware implants deployed have been distinct from one another, indicating the wide variety of implants available at the disposal of Lazarus. Additionally, we've also observed similar TTPs disclosed by Kaspersky attributed to the Andariel subgroup under the Lazarus umbrella, with the critical difference being the deployment of distinct malware. While Kaspersky discovered the use of Dtrack and Maui, we've observed the use of VSingle, YamaBot and MagicRAT. 

Dave Bittner: The White House has issued guidance for federal agencies' use of software security practices. The memorandum instructs agencies to obtain a self-attestation from software providers that their products are in line with NIST's security guidelines. The memo reads in part, ensuring software integrity is key to protecting federal systems from threats and vulnerabilities and reducing overall risk from cyberattack. The NIST guidance provides recommendations to federal agencies on ensuring that the producers of software they procure have been following a risk-based approach for secure software development. Federal agencies must only use software provided by software producers who can attest to complying with the government-specified secure software development practices as described in the NIST guidance. 

Dave Bittner: Chris DeRusha, federal chief information security officer and deputy national cyber director, said in a statement, the guidance developed with input from the public and private sector, as well as academia, directs agencies to use only software that complies with secure software development standards, creates a self-attestation form for software producers and agencies, and will allow the federal government to quickly identify security gaps when new vulnerabilities are discovered. 

Dave Bittner: On September 8, CISA released four ICS security advisories. Two of them are for industrial control systems, one for an MZ Automation server library, which received a CVSS score of 10. Another advisory addresses a vulnerability in PTC Kepware KEPServerEX (Update A) mitigations for heat-based buffer overflow and stack-based buffer overflow vulnerabilities. The other two are for medical industrial control systems Baxter Sigma Spectrum infusion pumps and Hillrom Medical Device Management (Update B). On September 13, CISA released five industrial control systems advisories for systems from Hitachi, Honeywell, Delta Industrial, Kingspan and Paradox. The most recent tranche of ICS advisories arrived on September 15. They numbered 11 and cover advisories for products from Siemens. As always, you can find the details on CISA's website, cisa.gov. 

Dave Bittner: Our guest this week is Rachael Conrad from Rockwell Automation. Our discussion centers on how industrial automation organizations can achieve their connected enterprise by providing a safe and secure infrastructure. Here's my conversation with Rachael Conrad. 

Rachael Conrad: Yeah, well, let's see - so I've been in the industrial automation space my entire career - and I hate to say how many years that is, but over 20 - and have had the opportunity in a variety of different roles to meet with probably 1,000 customers in the space over that amount of time. And, you know, I think one of the things that is probably the most challenging about the industrial automation and just manufacturing itself is - are the age of the assets are all over the place in plants. Some plants are brand-new greenfield facilities, but for the most part, they - you know, plants are a mix of new, old, in-between assets. And that's, I think, part of the biggest challenge for companies is to get their arms around the assets they have, where - and those assets really are part of the source of where risk can come from. But it's where they get the value out of, too. So it's interesting. 

Rachael Conrad: And, you know, I think what - a lot of times, you know, as cybersecurity and just network and infrastructure evolved in companies, the plant floor was really one of the last places that people looked, and people assumed that plant floors weren't connected. They didn't have connected assets. But obviously, with digital transformation and just the desire to get information and insights from assets on the plant floor, that's really not true anymore. There's so many assets that are connected. And while that's great opportunity to get valuable information to make really good decisions on, it also can be a risk. And I think that's really what's changed over time. And now we've got companies that are trying to figure out what to do, where to start. And they really don't know how to approach it on the plant floor as much as they do on kind of the IT carpeted space. 

Dave Bittner: Can you sort of paint a picture for us? I mean, when you look at that plant floor, I would imagine, as you describe, you know, there's such a range of ages of things out there. But I imagine also, over time, old equipment has been modified or additional things have been hooked up to them so that they can be monitored, so they can be integrated with some of the more modern conveniences. 

Rachael Conrad: For sure. So you might think of, like, a manufacturer plant - let's say, like a cookie facility that's making a bunch of different products. You know, some products and some lines have been there for longer. Some lines are new. It's a new product that they want to bring to market. And also, like, I think what's really interesting is that the equipment comes from a variety of different OEMs. Some of them have been integrated onto existing lines. Some of them, like I said, are brand new. And just understanding, like, what that infrastructure looks like and having a documented record of what assets are connected, just a lot of plants don't have that. So that's that digital record of what's out there and what's connected. 

Rachael Conrad: So you're exactly right. It's different ages of assets. It's different types of equipment that have been changed over time. It's, you know, maybe there was an ideal layout of the network infrastructure when the line started. But, you know, as it grew and changed over time, people added different assets, different monitoring, right? And there's not really a digital architecture of what that is anymore and how it's changed. That's very, very common. 

Dave Bittner: Is there a bit of a cultural element to this as well? I mean, I'm thinking of that - you know, that person who's been working down on the shop floor for decades and has all of this institutional knowledge that, as you say, may have never been written down. 

Rachael Conrad: Yeah - very, very true. That - you know, they know the sound of the equipment when it's running right, right? They know - you know, they know what the application should look like. And I think, also, the cultural side of it is, sometimes the plant floor has been really disconnected from the carpeted space and just the IT. Like, where IT would have concerns about things, the plant floor has just been doing it, and it's been working. So they haven't really thought about the security risk of their changes that they've made. And it's not bad intention. It's just they're doing what they need to get production out and probably don't have the background. They have the background in the automation. They have the background in the OT side of how to make the plant work, but they don't understand the cybersecurity side of it and those risks. So they don't think about it in that way. 

Dave Bittner: Yeah. That's a really fascinating insight - I mean, the notion that the IT and the OT folks may have very different ways of measuring success. 

Rachael Conrad: For sure. And I think, you know, one of the things that we find the most successful when we're going out and talking to OT customers is getting the IT and OT people in the room to talk about it, and, you know - 'cause what you don't want is there's things that you might not think about in IT when you're putting controls around to protect your carpeted space. They're not the same things that are important to a plant that has to run 24/7. You can't go down to patch things. You can't - it's just different frequencies. It's different things that IT might not know about OT and that OT might not know about IT. So just that - brokering that change conversation and that discussion about, this is what we need, this is what we need, and bringing those two groups together is typically the foundation of getting something really great started and moving forward. 

Dave Bittner: So what do you see - you know, as you're out and about, as you say, working with the organizations that you do, are there any common things in the organizations that are successful here, that are kind of leading the way? 

Rachael Conrad: Yeah. You know, when we take customers through a journey here, you know, most manufacturing, and I'll say it, you know, at least two-thirds of manufacturers from the customers that I visit haven't really started on their journey. But we use the NIST framework for first - like, the first step in any journey in cybersecurity is identifying the assets. Before you can detect or prevent cybersecurity breaches or respond and recover, you need to identify what's there so you have a good digital picture of where all the risk is. And so that's where we start. I mean, it's really simply, let's put up, let's say, a software in place that just goes out and identifies all the assets that are connected to the network. It's not intrusive. It just goes out and crawls the network and basically builds a digital print of what assets are connected. And that's where you start. And that's where you start to say, OK, now that we know what assets are connected, we can start to say, what do all of these different types of assets, what kind of threats do they bring? And then you build a plan around that. 

Rachael Conrad: So you build a plan to detect threat. So you might use a platform - a continuous threat detection platform. There's a lot of different ones on the market. We partner with a lot of different - a couple different technologies there - Rockwell when we're delivering services, but - so putting a platform in place. But then once you put this continuous threat detection platform in place to continually monitor the threats, what happens next is these threats become - or these alerts and alarms become overwhelming in the OT environment because nobody has the context of what they mean. So you got to have somebody with both the OT expertise and a knowledge of cybersecurity to help you tune those - tune out the noise - take away the alarms and alerts that really don't mean anything - that aren't going to cause any problems - and really focus in on the ones that are going to be the problem for you. And that's just the next logical step that we walk customers through. 

Dave Bittner: When you're presenting to someone who may be behind on this sort of things, I would imagine - I mean, is it important that you have to sort of break it down into smaller pieces so that it doesn't seem like it's overwhelming? 

Rachael Conrad: Yes, exactly. Because just even, like - I always say progress over perfection, right? Getting to - like, the end game is that you have this adaptive cybersecurity program. You know, you have all the - you know, you're utilizing a secure operation center that's monitoring in real time, fixing things in real time, but you don't start there. You start with the simple basics and just segmenting your network, assessing your assets, maybe protecting your endpoints, and then you just move along at the pace that's right for you. And what that pace is is how much risk do you have, what are you - what is the assets that you're trying to protect? Is it intellectual property? Is it people assets, right? Is it some sort of secret recipe that you don't want anyone else to know about? It really depends on how much - what your risk profile is. And then we help you figure out how to - what to invest over time to protect that risk. 

Dave Bittner: How do you go about getting, you know, buy-in from the various stakeholders? 'Cause I'm thinking, you know, even - from the CEO of a company to the board of directors on down, all the way down - you know, the person sweeping the shop floor - they all have different levels of being on board or not, and they could slow things down if they're not on board. 

Rachael Conrad: Sure. And I think there's a lot more stakeholders and decision-makers today in the industrial world than ever before. But I think it really starts with painting a picture of the vision of where you want to go - the end game, right? Start with the end game in mind. Like, this is why it's critical. These are the risks that we're facing, and here's where we've identified there's gaps in our security compliance. And we understand it, and here's our plan to get there. But we're going to do it in an approach that we're going to take some action, and then we're going to test what that action - is it doing what we said? 

Rachael Conrad: So just like any sort of business case that has a return on investment - about quantifying - what does it cost you for an hour of downtime if you have a security breach? What does it cost you for a week of downtime if you can't get back up and running? And we work with our customers and clients to just build that risk profile, and that's how we help them sell it to their leadership teams. It's like, we can't afford to go down. Or if one plant goes down, it's connected to all of these other plants, and here's how much it costs us. So this whole idea of getting back up and running - 'cause, look, you're going to have some sort of cybersecurity incident. 

Rachael Conrad: You know, that - you can look at many, many studies. And even in the last - I think there was a LNS report that said 53% of industrial manufacturers have experienced a cyber breach in the facility over the last three years - so half of the facilities. That's big. And maybe it's a little breach. Maybe it's one that is in the news and that brought down 15 plants across the country, right? You don't want it - that's not the news that you want to be in. You don't want to be that customer, so - and that's huge dollar loss for customers and manufacturers. So that's really what it's about. It's about how much - what do we have to do to get the basics in place to just minimize that risk and then maximize our time to get up when it does happen? Because it's going to happen. 

Dave Bittner: Have we reached the point where folks are realizing that there may be a competitive advantage of being on top of this stuff? 

Rachael Conrad: Yeah, I mean, I think so. I think about, you know, just, like, the past couple years during the pandemic, and I think, as people move to more remote monitoring of applications and just remote operations of their plants, I've never seen so many customers interested in and really talking about cybersecurity. Either they're recognizing more or they're saying, well, we want to have supply chain resiliency for our customers. So everything that we can do to keep running is a competitive advantage in how we sell our products in the market. 

Dave Bittner: Our thanks to Rachael Conrad from Rockwell Automation for joining us. 

Dave Bittner: In this week's Learning Lab, Mark Urban has part one of a two-part segment exploring the scale of the generation of electricity. Here's Mark Urban. 

Mark Urban: Thanks. And hello again. I'm Mark Urban. Last episode, we explored with Miriam Lorbert how control loops manage specific processes to accomplish tasks with precision. As a core component of industrial systems, the concept of a control loop can be applied to many different variables, many different inputs and, you know, many different applications. But when you chain them together over and over and over again, you can create a massive scale in the industrial systems that we see today. Now, electricity is probably the most expansive and massive industrial system that I could imagine. Oil and gas may be close. There may even be more. But I wanted to focus on electricity in two parts. 

Mark Urban: First, in this episode, we're going to explore the sheer scale of the generation of electricity. It's control loops times, you know, thousands and millions. In the next episode, we'll be joined by Dragos' own Phil Tonkin, a veteran of the electrical distribution industry. He'll help us learn how transmission grids work, how electricity goes from power plants to your home. But we're going to focus this on that kind of power plant or that production side of it. 

Mark Urban: And I wanted to step back and start with just the sheer scale that we see in the electrical utility industry. In 2021, globally, across humankind - generated 28,466 terawatt-hours of electricity. Now, China produced the most - about 30% of the total, 8,634 terawatt-hours. The U.S. was second at about 15%, followed by India at 6%, then Russia, Japan, Brazil, Canada, South Korea and Germany. I think what I found interesting was that three countries account for over 50% of global production and usage of electricity. China was almost double the U.S. in production, and the U.S. was about 2.5 to 2.6 times what India produced. Now, that measurement was a terawatt-hour. To understand that a little bit better, I want to go down to the basic unit, or the watt. And the watt is used to measure the rate of electrical transfer. It actually came from a guy named James Watt, or named after him, an 18th-century Scottish inventor who worked on steam engines. And apparently, many of his works were fundamental to the industrial revolution. So he knew electricity. He knew power. 

Mark Urban: It also - another way to measure it is one joule per second. There are a lot of ways to express it involving Newtons or ohms, volts and amps. In fact, we see volts and amperes as common measurements on things like cellphone chargers and computer power supplies. But let's put it in terms of how people use it. A cellphone charger uses about 10 watts. A microwave uses somewhere between 650 watts and 1,800 watts. And building up from watts, a kilowatt is a thousand watts, a megawatt, a million watts, a gigawatt, a billion watts, and a terawatt is one trillion watts. And so a terawatt-hour is a unit of energy equal to outputting 1 trillion watts for one hour. Now, if we said that world production was 28,466 terawatt-hours, that's 28,466 trillion watts for an hour. 

Mark Urban: All right. So those are big numbers. It's really hard to imagine that scale. So let's bring it down to - let's look at a microwave. A typical microwave, the one I have at home, is about 1,000 watts. Some are more. Some are less, as we said. If you ran a 1,000-watt microwave for an hour, it would use 1,000 watt-hours. So then if you ran 28 trillion microwaves for an hour, that's pretty much how much electricity is generated and consumed across the world. That's how much. 

Mark Urban: Now let's turn to, where does it come from - 'cause electricity is generated from different sources. More than 95% of it is stuff that turns a turbine that powers a generator. And a turbine is like a fan or a jet engine. You know, it's got the blades. Now, wind can turn a turbine. Water can turn a turbine. The sun can heat water to create steam to turn a turbine. That's known as solar thermal - solar thermal, rather. You can burn coal or natural gas or oil. The gas produced can turn a turbine. And the heat, you can heat it to make steam to turn a turbine. So that's - you know, a lot of it is about turning turbines. Now, let's do what happens with that turbine. It powers a generator. And the generator works by rotating a coil of copper, the conductor, through a pole of a magnet - through each of the poles of a magnet. So the electricity - the result is electricity. 

Mark Urban: So fuel or some power, turbine, generator, electricity - it's science, even if it seems like magic. There's one form of energy that skips the turbine, by the way, and that's solar that uses solar - or photovoltaic - there's one type of solar that bypasses turbines, and that's the solar we know using photovoltaic cells that convert sunlight directly into electricity with no turbines involved - so spin a turbine, generate electricity or not, if you have photovoltaic cells. 

Mark Urban: Now, that's grossly simplified, right? There are dozens or hundreds of small processes that happen in the course of turning turbines and generating electricity across all of those different kind of fuels or ways to turn the turbine. And that's just the generation part of electricity. We haven't even touched on the transmission of electricity - getting it from the generation plants to your microwave, to your phone charger - and that's the grid. And that's what Phil's going to help us understand next week 'cause that's a huge, big, complex set of systems. But let's keep on generation for a bit because it's - electricity is generated by power plants, generation plants. And there are about 35,000 power plants across the globe. You've got very large ones, and you've got very small ones. The largest one in the world is of Three Gorges Dam, and that's using hydroelectric energy. And it produces about 112 terawatt-hours of electricity in the year 2020. So that's what it was measured at. That's the biggest. 

Mark Urban: And if we want to talk about average, we first have to look at the core capacity in these in these plants. So we talked a microwave consuming a thousand watts - a kilowatt. Power plants have a capacity usually expressed in megawatts - millions of watts. In power generation, capacity is the amount of electricity a generator can produce when it's running full blast. So the actual output depends on how long it runs and how close it runs to its max. Our Three Gorges Dam example has a capacity of 22,500 watts. If it ran at that full capacity 24/7, 365 days a year, it would generate over 197 terawatt-hours. In reality, in 2020, it put out about 112 terawatt-hours, so that's about 56- to 57% of the theoretical max. 

Mark Urban: So capacity is a max, and it doesn't always run at the max. And there are reasons for this because you need maintenance of these complex systems, so you need to shut them down for periods of time. You may have demand changes, so if aggregate demand falls - let's talk about at night in your sleep - you shouldn't produce it, right? Electricity generally has to go somewhere eventually. Another reason is variability in the inputs. 

Mark Urban: Renewables, like wind and solar, are highly variable. Let's take a look at a windmill. A windmill has a capacity of 10 megawatts - that's among the biggest. But enough wind needs to be blowing on a constant basis to produce that max electricity, and that's not always the case. Even hydropower can be variable. In fact, the Three Gorges Dam, the largest producer in the world that we talked about, is having a tough year due to drought on the Yangtze River. So there's less water, less turbine turning, less power, almost 50% less by some accounts because of the levels of the water. 

Mark Urban: All right. So that's a little bit about the power plants, 35,000 of them, the capacity. Now, what makes up those? What's the largest sole? Definitely not hydroelectric dams - hydroelectric is up there. But the largest sole - you know, let me back up. Now, let's turn for what creates the power. Hydroelectric is big, but it's definitely not the biggest. The largest source of power generation is coal. It accounts for about 36% of global production - burn coal, make steam, or use the gas, to turn a turbine. The next largest is natural gas at just over 23% of global production. Hydropower, water, comes in at third, about 15%. Nuclear is just shy of 10%. Wind is now about 6.6% - solar, 3.7%. Then there are other renewables at about 2.7%, with oil and gas about 2.6%. 

Mark Urban: Now, solar and wind are on the rise big time, but it's off of a very small base relative to these other ways of generating power. If we look at China, for example, coal is by far the No. 1 source of energy production. Solar and wind are climbing massively but from a very tiny installed capacity. Coal generation is still No. 1 in - I'm sorry, coal generation is still growing in China. It's five to six X more than solar and wind combined. In the U.S., by contrast, gas is No. 1, followed by coal and then nuclear. And the amount of electricity from coal in the U.S. has declined significantly over the last few years. It's a contrast to China where the capacity is still growing, or the amount of energy produced by coal is still growing. In India, third-largest producer, coal is the biggest source, and coal is still growing there, even as solar is skyrocketing. Germany, coal is No. 1, but wind is No. 2 after coal and growing very fast. And the U.K., No. 1 source of electricity is gas, the wind is also growing very fast since it already reached the No. 2 position. France has been dominated by nuclear, so the transition to solar and wind has been much slower. OK, so those are the sources. 

Mark Urban: We - just to back up, because I just went through a lot of stats, let me summarize those on the generation side. Massive industrial scale, 35,000 or so generation plants across the globe, making 28,466 terawatt-hours of electricity. It seems - it's daunting. It's massive. It's solid like a mountain or a chain of mountains. But electricity flows. It's dynamic. You know, the next time, in the next episode, we'll explore the transmission, the grid. We'll look at the mechanisms that manage the flow of electricity, the ebbs, the flows of demand, the delicate balance that the grid achieves and sometimes barely achieves or not quite. So hopefully, you'll come up with a bit better understanding of electricity today. And stay tuned in a couple weeks, and you'll find out how it moves to you. Until then, I'm Mark Urban with Learning Lab. 

Dave Bittner: And that's "Control Loop," brought to you by the CyberWire and powered by Dragos. For links to all of today's stories, check out our show notes at thecyberwire.com. Sound design for the show is done by Elliott Peltzman with mixing by Tre Hester. Our senior producer is Jennifer Eiben. Our Dragos producers are Joanne Rasch and Mark Urban. Our executive editor is Peter Kilpe. And I'm Dave Bittner. Thanks for listening.