Preparing for the electrical grid of the future.
Dave Bittner: It's November 30, 2022, and you're listening to "Control Loop." In today's OT cybersecurity briefing, the U.S. Government Accountability Office issues a report on offshore oil and gas cybersecurity. The Oak Ridge National Laboratory seeks to secure power grids. Boa Webserver vulnerabilities are used to target energy organizations. CISA updates its infrastructure resilience planning framework and issues advisories for ICS vulnerabilities. Our guests are Mara Winn and Guohui Yuan, joining us from the U.S. Department of Energy. Mara and Guohui discuss their report, "Cybersecurity Considerations for Distributed Energy Resources on the U.S. Electric Grid." On the Learning Lab segment, Mark Urban is back and has Dragos' CISO, Steve Applegate, with him on starting an OT cybersecurity program.
Dave Bittner: The U.S. Government Accountability Office has published a report reviewing the cybersecurity of offshore oil and gas infrastructure. The GAO recommends that the Department of the Interior's Bureau of Safety and Environmental Enforcement immediately develop and implement a strategy to address offshore infrastructure risks. The GAO says such a strategy should include an assessment and mitigation of risks and identify objectives, roles, responsibilities, resources and performance measures, among other things. And the report also says the Department of the Interior has generally been receptive to its recommendations. The GAO notes that the BSEE says the severity of cyberattacks could be mitigated by manual overrides, but the report adds that BSEE officials were not aware of any assessments confirming that manual controls could mitigate the impacts of cyberattacks. The GAO points to the 2010 Deepwater Horizon disaster as an example of an incident where even manual safety systems failed, though this event wasn't caused by a cyberattack. The report finishes stating BSEE has struggled to address cybersecurity risks to offshore oil and gas infrastructure and only recently has taken steps to start a new initiative.
Dave Bittner: This effort remains in the earliest stages of development. Accordingly, it is not guided by an overarching strategy that identifies cybersecurity risks, relevant practices to address those risks, the bureau's role in addressing them, milestones for activities such as formalizing relationships with other federal agencies and industry organizations, resource needs, such as appropriate staffing levels, and performance measures to assess results.
Dave Bittner: The U.S. Department of Energy's Oak Ridge National Laboratory is researching ways to use high-fidelity sensors and blockchain technology to secure electric grids against cyberattacks. The project, dubbed DarkNet, is focused on securing grid equipment and communications. ORNL stated DarkNet researchers are developing a private network architecture that grid operators can scale up and use to quickly and accurately control power generation and transmission equipment that may sit hundreds or thousands of miles away from a central operational control center without fear of cyber intrusion. The scientists are testing the architecture on ORNL's own grid equipment. Next, they will demonstrate communication on a regional scale and, later, on a national scale.
Dave Bittner: Microsoft has expanded on an attack earlier described by a Recorded Future back in April, in which Chinese state-sponsored actors targeted Indian power grid organizations, an Indian National Emergency Response system and the Indian subsidiary of a multinational logistics company. Microsoft says the attackers exploited Boa, an open-source web server that was discontinued in 2005. The researchers note that Boa is still used by different vendors across a variety of IoT devices and popular software development kits. Microsoft determined that all of the IP addresses published as IOCs by Recorded Future were connected to Boa servers, and half of these IP addresses returned suspicious headers.
Dave Bittner: The researchers explain, investigating the headers further indicated that over 10% of all active IP addresses returning the headers were related to critical industries, such as the petroleum industry and associated fleet services, with many of the IP addresses associated to IoT devices, such as routers with unpatched critical vulnerabilities, highlighting an accessible attack vector for malware operators. Most of the suspicious HTTP response headers were returned over a short timeframe of several days, leading researchers to believe they may be associated with intrusion and malicious activity on networks.
Dave Bittner: Microsoft found that there are currently over a million Boa servers exposed to the internet, the majority of which are located in India. The researchers conclude the popularity of the Boa Webserver displays the potential exposure risk of an insecure supply chain, even when security best practices are applied to devices in the network. Updating the firmware of IoT devices does not always patch SDKs or specific SOC components. And there is limited visibility into components and whether they can be updated. The known CVEs impacting such components can allow an attacker to collect information about network assets before initiating attacks and to gain access to a network undetected by obtaining valid credentials. Microsoft adds that this reconnaissance is particularly important when launching attacks against ICS environments, stating, in critical infrastructure networks, being able to collect information undetected prior to the attack allows the attackers to have much greater impact once the attack is initiated, potentially disrupting operations that can cost millions of dollars and affect millions of people.
Dave Bittner: ESET reports a surge in a ransomware variant the company calls RansomBoggs. Deployed against Ukrainian targets, the malware is written in .NET and represents a new strain of ransomware. But the deployment, ESET says, is similar to what they have observed in Sandworm activity in the past. Sandworm has been associated with Russia's GRU. The researchers tweeted, there are similarities with previous attacks conducted by Sandworm. A PowerShell script used to distribute the .NET ransomware from the domain controller is almost identical to the one seen last April during the Industroyer2 attacks against the energy sector. ESET also sees similarities between RansomBoggs and IRIDIUM, Microsoft's name for the GRU operation the company detected in Prestige ransomware attacks against Polish and Ukrainian targets in October.
Dave Bittner: CISA has released an updated version of its Infrastructure Resilience Planning Framework to help state, local, tribal and territorial planners. The new version of the IRPF includes a new tool for identifying critical infrastructure, the Datasets for Critical Infrastructure Identification Guide. This data set provides users with guidance on how and where to find publicly accessible geospatial information systems on critical infrastructure assets via the Homeland Infrastructure Foundation-Level Data site as well as several other GIS sites. It provides guidance on the challenges of getting a diverse set of opinions when planning. It can be challenging to get all the right stakeholders together and ensure that a diverse range of opinions and interests are considered.
Dave Bittner: The IRPF 1.1 expands on the process of gathering stakeholders. It provides new drought resilience information via CISA's National Drought Resilience Partnership. This includes a new guide that provides an overview of the drought hazard, examples of direct and indirect impacts it can have on infrastructure systems and federal resources for assessing and mitigating drought risk. And it includes revised resilience concepts that incorporate CISA's methodology for assessing regional infrastructure resilience. It provides additional details on analytic methods that planners can use to improve their understanding of infrastructure systems in their community. On Tuesday, November 29, CISA issued a number of ICS advisories. As always, check CISA's website for the latest information.
Dave Bittner: Mara Winn is deputy director for preparedness policy and risk analysis at the Department of Energy's Office of Cybersecurity, Energy Security and Emergency Response. Guohui Yuan is program manager for the Department of Energy's solar grid integration research and development. I spoke to both of them about the DOE's recently released report "Cybersecurity Considerations for Distributed Energy Resources on the U.S. Electric Grid."
Mara Winn: So I sit in the Department of Energy's Office of Cybersecurity, Energy Security and Emergency Response. And when we look at how the energy system across the U.S. is going to change, we see those - rapid transformation of the electric grid with an increasing number of distributed energy resources. We know that Americans' produce and consume energy is shifting, and we need to embrace a cleaner, more efficient, sustainable future. So as these distributed energy resources are integrated on the grid, it's critical we continue to maintain the reliability that customers expect. And this means preparing for all hazards, including cybersecurity, natural disasters and other physical security events.
Dave Bittner: Well, Guohui, can you explain to us what exactly we're talking about here with this transition? I mean, what in the most recent years is shifting from how the grid has traditionally run?
Guohui Yuan: Sure. So there are two megatrends that are converging together here. On the one side, because the decarbonization goals of the administration, we're deploying a lot of the renewables, wind and solar. And on top of that, some of these technologies are connected to the distribution grid. We call them distribution energy resources or DERs. So that's one trend. The other trend is that the grid is undergoing a massive transformation by integrating a lot with digital technologies, communications data as well as the automation controls. So those are coming together, and it makes the planning and operation of grid a lot different from the traditional grid.
Dave Bittner: And where do we stand today in terms of executing this transition? I mean, there's certainly been talk for the past few years about things shifting online and some of the cybersecurity concerns. What's the lay of the land today?
Mara Winn: I'll jump in first with that. With the bipartisan infrastructure law and the Inflation Reduction Act, the Department of Energy alone has $62 billion that it's directing towards clean energy transition currently in motion across all 50 states. So that is a significant influx. We are seeing the focus on resilience and clean energy starting to enter the grid, starting to be a significant topic of conversation, even with our utility partners - not just those on the ground doing the direct DER work, but those that know that this is going to integrate into the grid. And part of the work that we're doing here is to make sure the conversations are starting early. We need to do security by design. We need to make sure that these energy resources are entering our systems in a safe and secure way that not only takes into account their individual risks, but how they're bringing their risks and connecting into the greater nation's infrastructure.
Guohui Yuan: Just to follow up on that, so on the technology side, there are many different technologies that we're talking about here. So when we talk about DER, we're really talking about solar. We're talking about energy storage, electric vehicles, buildings and so on and so forth. So this diverse of the technologies actually are making the system work in cybersecurity defense and the grid operation more complex. And then the one other purpose of this report is to raise the awareness that Mara was talking about.
Guohui Yuan: First of all, the different industries have a different perspective of what cybersecurity is. Even though there are many standards that are out there, they're not consistent sometimes, and a lot of the standards need to be refined and need to be harmonized. For example, on the DER side - solar side - there is a 1547-3 standard. Then you have the - in the bulk power system, you have a so-called CIP, which is a critical infrastructure protection standard that's to protect a bulk power system. And then you have an EV standard and then so on and so forth. So these standards can be confusing at some times, and they need to be harmonized. And a lot of that is awareness and education.
Mara Winn: And we also recognize that people are starting in different places. You have some entities that are very sophisticated - they understand the nuances - and we want to make sure we're meeting them where they are. But we also understand that there's a lot of entities in this mix that just haven't even approached this. So making sure adopting the best practices and meeting minimum security requirements are part of the conversation. Because of that diversity of different types of energy resources, we know that we have to cast a greater net to make sure that we are integrating with lots of different stakeholder communities that may not have been part of the energy security conversation in the past, but we know we need to bring them into the fold, support them, make sure that they have the resources to meet the needs of our future energy infrastructure.
Dave Bittner: Yeah, you know, you talk about resiliency, and I think that's really striking in that, you know, my understanding is that our energy grid tends to be kind of regional. Are we talking about more connectivity between those regional players? Is that part of the future of our grid?
Guohui Yuan: Let me answer that first. The future - that's one future of the grid. And if you look at the - today's grid is already connected at the national scale. So when we are talking about the future grid, it's more at the distribution level, where a lot of technologies are happening and being deployed. So that's why we're focusing on DERs. And the other aspect of this is that the DERs are usually not owned by the utilities and the system operators, and on the other hand is they are connected to the grid. So they have impact to the grid, but they're not, you know, monitored or controlled by the system operator. Therefore, it presents additional challenges for ensuring the grid reliability and security.
Mara Winn: Yeah. And I'll also add - because I love that word resilience, right? And when we think about resilience, we think about adding in a lot of these resources into our grid to give us resilience so we don't have a single point of failure. We think about the hurricanes that come through. We think about other kinds of natural disasters. As climate change continues to evolve, we have to be prepared for that future state that we're in, and we need resilient solutions to be part of that. And DERs are a key component of it, but we also need to make sure that, implementing cybersecurity requirements, grid and DER planners should build cyber defenses in with the goal of surviving an attack while maintaining critical functionality. That's also resilience. We need to make sure, when we're trying to solve one problem, we don't create another vulnerability in the grid.
Dave Bittner: You know, one of the things, as I read through the report, that comes up over and over again is the notion of partnership and how important that is at the federal level, at the local level, the providers and even the consumers themselves. Can you speak to that some, Mara? Can I start with you? I mean, this is - this really is a team effort, here.
Mara Winn: It really is. You know, historically, many DER partners have not been part of our prior partnerships with the Department of Energy. And also, the oversight that operated and maintained the power grid, the electric power reliability and security requirements and responsibilities haven't been part of the conversations. But we need to bring them into that conversation. We've been doing a lot of outreach with the Department of Energy, making sure that we're meeting those stakeholders where they are. And it also requires a different approach than you would take with some of the major utilities because you have people meeting in different places, communities forming in different environments. That's one of my favorite things about our partnership with Guohui's office is they are on the ground doing the work in the development, and we're able to fold in all of those energy security and cybersecurity conversations to the design effort because that's - we want to meet the communities where they are. We know that we have to think more flexibly because it's not one single entity and ensure that we are appreciating the challenges of these communities.
Guohui Yuan: Yeah, just to follow up on that, we really appreciate the partnership with CISA office that Mara represents. A big part of the stakeholder engagement is education. We want the different communities to understand that there are resources available to them a lot of times that they don't know they're there. And they just are confused or overwhelmed at the different information. I've mentioned standards, and there are other informations (ph) as well that's out there. And they don't know which way is the requirement and which way is nice to have. So a big part of it is the education.
Guohui Yuan: So we do that through collaboration with the National Labs. And actually Sandia, NREL were part of the report, supporting the report, as well as working with the communities. I give an example that we're working with the National State Energy Offices and the national regulatory commission, called the NARUC, to really spread word, educate them as well, so that the decision-makers can understand the challenge of cybersecurity and what are the tools available to them. There are a lot of people involved, and we need everybody to really pay attention to cybersecurity challenges and know where the resources are and work with us.
Mara Winn: Yeah, that's an excellent point because we need to make sure good governance is implemented - designing security into utility and DER systems from the beginning and making security a priority for all the employees, suppliers and customers. It needs to be part of the regular conversations. But also incentivizing cyber resilience - we talk about some of the regulations that go into our energy systems. And the Department of Energy is not a regulator, but we are making sure that those who are regulators are knowledgeable on what the challenges are and what practices need to be in place. We need to make sure that cyber resilience is incentivized, and it goes beyond the standards and work to actively detect threats and adopt a zero-trust approach to verify commands and data. We need to make sure that it is a comprehensive system and people understand the whys, as Guohui was explaining, so they can take the proper actions.
Dave Bittner: Yeah. It strikes me that, you know, obviously there are challenges ahead, but this also presents a tremendous opportunity here, especially, as you mentioned, you know, we've - we're in a position where there is some good funding available. Are the two of you optimistic as you look ahead towards the horizon?
Mara Winn: I am very excited. I think that there is a significant influx of funding from all different sources and even within the private sector community. I had the opportunity to go to the RE+ conference in September and see a lot of this work on the ground. And you see the excitement, and you see the camaraderie of these communities coming together to drive a lot of change. And I think that that funding conversation needs to also make sure that we are discussing the cybersecurity practices as part of it. We can't design blind to them and then expect us to have a resilient and reliable system that we expect from our infrastructure today.
Guohui Yuan: I would agree. And just to give an example about the excitement, not only at the DOE level and National Labs - recently, I supported - we supported CESER in this cybersecurity competition. This year, the theme is solar. And organizers develop a scenario where the solar powered EV manufacturer was compromised and then what the solutions could be. We have more than 100 teams that are joining - college teams that are joining the competition. There's, like, seven in there. And there's a - we know there's a lot of next-generation cybersecurity experts that are coming out of the pipeline. So I'm very excited about that.
Mara Winn: Yeah, that's a great point because we need to make sure when we go work with these organizations, with these companies, with these businesses and tell them the requirements that you also have the people who are trained and understand, and it's not just an IT problem. It's an OT problem as well and making sure that that excitement exists within those who are in the learning communities. CyberForce is a great program with the collegiate community. It's a great competition to drive a lot of that excitement. And we try to make sure that it's relevant to today's issues. And so, as Guohui said, it talked about a solar farm that was tied into an electric vehicle charging station, and you had students working to defend it and working with our best red team experts across the National Labs, so they are ready to take on these challenges in the world that we're building.
Dave Bittner: Our thanks to Mara Winn and Guohui Yuan from the Department of Energy for joining us.
Dave Bittner: In our Learning Lab segment, in part one of a two-part series on starting an OT cybersecurity program, Mark Urban speaks with Dragos' CISO, Steven Applegate.
Mark Urban: Hi. Mark Urban with the Learning Lab. Today we're going to get a CISO perspective on building an OT security program. And I'm fortunate to be joined by Steve Applegate, the CISO here at Dragos. Steve, welcome.
Steve Applegate: Hey. How's it going, Mark?
Mark Urban: It's going well. So, you know, we actually got talking at the lead-up to DISC, the recent Dragos Industrial Security Conference, that we held just outside of Baltimore. And you and I started talking about you've been in a couple different places overlooking security, helping to make that transition to OT security and so came up with the idea for this episode to talk through some of those observations and some of those experiences that you've had. So I wonder - maybe you can start out with just a little bit of background and how you got here, Steve.
Steve Applegate: Sure. Yep. So I started my IT career back in 1989 when I was assigned by the United States Air Force to work as a computer specialist at NORAD Cheyenne Mountain Air Force Base. I continued on in DOD for probably about 10 years, 11 years and - in various roles, starting as active duty, and then I ended up as a contractor. And, you know, my first experience with OT actually happened during that 10 years. I was - I joined a contract in Iceland where the company I was working for was developing a very early kind of - you could call it rudimentary distributed control system. And I got to do some, like, real nitty-gritty device diver - device driver programming. And I was - you know, it was kind of an additional duty while I was also supporting a bunch of developers and doing sysadmin and network admin kind of work. And I continued on in - you know, in a very highly technical roles over the years. Probably the first half of my career, you know, was technical. And then over time, I found myself in leadership roles, increasingly.
Steve Applegate: My last about eight years, I've been in leadership, starting with Saudi Aramco, where I was helping - their first CISO ever, like, was building a program and was taking over the, you know, organizing and centralizing all the security functions into one group. And I got to kind of help, you know, work with all these different stakeholders and - especially on the OT side and some really great people, like, pulling together, you know, all the initiatives and organizing it all. And then I went from there - followed that gentleman back to Marathon Petroleum, where I was kind of in an acting CISO role and doing some work with - as they were doing a big merger and acquisition and stuff. I left Marathon a couple of years later and went to PepsiCo, where I was the deputy CISO for the global enterprise there and probably about 500 plants, if you - depending on how you define a plant. Very heavy - you know, I was a stakeholder on the OT side and got to do a lot of really cool, early work with helping to accelerate that program. And then two years ago, roughly, joined Dragos as the first CISO here. I guess that gets you up to speed on my life.
Mark Urban: All right. Well, that's a pretty deep experience base. You know, as you - you talk about a number of stops and, you know, some of the projects, especially as you kind of aggregated experience. If you were to then look at when you're making that transition to looking at the OT security problem set, what - you know, what are your observations on, you know, the keys to - 'cause a lot of the people that we work with, a lot of the companies that we work with tend to be - you know, have fairly mature IT organizations from a security perspective, but OT has lagged behind the investment curve and only now is getting a tremendous amount of attention. And I think you've been in situations like that. So I wonder, you know, as you make that jump to looking at building that OT security program, you know, what are the - some of the observations that you make about, you know, key steps?
Steve Applegate: Oh, that's a really cool angle to take the conversation. I know, you know, there's a credibility gap when people from IT or people from the business, you know, from a non-OT background come in and try to start dictating how things have to be, and they try to take some of those really key learnings that we've had in IT for decades, literally - IT security programs, and they try to apply them directly, or they try to shoehorn them in. And you mention the OT problem. I mean, it's absolutely a different problem and a different set of circumstances and a different set of risks. And, you know, even the prioritization - you know, like, OT security 101 looks at, you know, turning upside down the CIA triad and says the most - the first thing you got to look at is availability.
Steve Applegate: So you flip it over and, you know, a lot of times IT people - meaning well, of course, I think - I always kind of, you know, give people the benefit of the doubt, but they try to take these - the IT mindset in, and they lack that credibility with OT. The solutions they offer are wholly - you know, wholly wrong and even dangerous or harmful. You know, and I can go back to my practitioner days, and I remember very clearly when I made a mistake like that. And I - I've tried to slap an - you know, an old-school antivirus solution onto a control system, and I ended up forcing a failover. Luckily, we didn't have a true outage because of it. But I learned a very early lesson in my career that you can't just take IT mindset and just make it work in OT. You know, a lot of people - or I've heard it said before that governance - you know, if you go back and look at the actual - the cause of any breach or any security problem you have, you could point at - eventually, if you look back for the root cause, you could say there's a governance breakdown of some sort.
Steve Applegate: You know, I'm not sure that you could say every single problem could could be blamed on governance. But, you know, I know that, you know, governance plays such an important part. It's so foundational, especially in OT, where you have such a complexity. You know, you've got different leaders, different chain of command, different mindset, different language, you know, different vernacular. And to try to, you know, overcome that with loosey-goosey kind of governance of just, hey, you know, I know so-and-so over there. I'll give them a call - you know? - and there's no written processes, and you try to somehow make that work. Sometimes those trust relationships will help to get you to a point, but eventually there's going to be a governance breakdown.
Steve Applegate: And I just feel like, you know, with that balance, try to show them that you care, you feel their pain, you understand all of the, you know, critical systems that they run, and you have the same emphasis that they do or the same, you know, priorities. Try to find the right fit so you don't immediately jump to super mature processes that can't be maintained once you put them in. You know, there's that inflexibility if you jump to too much maturity, and then you slow down the business. You create, like, analysis paralysis, and those early losses like that can end up killing a whole program. So it's super important to find that good balance of, you know, true risk-based controls that are at just the right level so that you don't end up with a fragile environment that doesn't survive the test of time and stuff.
Mark Urban: Don't get too fancy, too complex out of the gate because that's hard to sustain?
Steve Applegate: Absolutely. Everything has to be built to sustain. You know, when you first go into the build portion, and you say, what are we going to put in? You have to say, OK, who can maintain this when we're done? Because if you're missing - like, missing head count, missing budget dollars, even just the cycles, you know, those plant people - a lot of times, plant people, you know, that are full time E&I techs or, you know, engineers working on a plant - in their spare time, they're going to operate a lot of the security components that we put in. So, you know, you have to try to understand what the true support needs are - what the level of effort is required to keep it evergreen. Otherwise, you put something in that actually will hurt your program instead of helping it.
Mark Urban: That's right. It's like - you know, one of the things that has become clear to me, especially through this podcast, is industrial systems are so much about locking in, you know, specific ways to do things and then just repeating it, repeating it, repeating it at, you know, massive repetition and scale and that, when you change processes, those are significant events as well that require a lot of planning - you know, maintenance outage. And so I think what I'm hearing you say is, you know, industrial processes are built a certain way to keep them relatively simple so that you can be repeatable, you know, in an automated way. And as you change, you have to take a lot of precautions and a lot of preparation to then change those capabilities. And I think that serves your point of - you know, keeping things simple to, you know, be able to maintain that consistency seems to fit very well with that environment.
Steve Applegate: I really like that, yeah. There's a lot to unpack in what you said. You know, these people that are maintaining the systems in a plant, in an OT environment, you know, they're actually, you know, they're pushing buttons that cause fire explosions to happen, opening valves, closing valves, doing all kinds of supercritical stuff. And if we overcomplicate the security, you know, and it takes resources away from, you know, their actual process control, not only is it not sustainable - again, that's where I use the word dangerous, you know? It also can - you know, so much of a company that has a big OT presence. So much of their revenue is dependent on those - everything happening in a plant - every process being perfect every time. Deterministic protocols - you know, you miss one ping on one system, it could be catastrophic. And, you know, something might not happen at the exact right time and - I don't know.
Steve Applegate: I don't want to be too grandiose, but, you know, a lot of the - you know, what's a worst case scenario? Somebody could die, or there could be a chemical spill that actually, you know, pollutes a water system or, you know, a watershed or something. So the stakes are so high that keeping it simple, I think, is just critical. And not even just simple, but keep, you know, securing at the right level. Like, whenever - you know, I was working at an electrical utility whenever NERC CIP was first coming about. And we had all kinds of great meetings and talking about - how can we meet these requirements and things? And a lot of times, we would want to - we get, you know, a bunch of engineers in a room, and we'd all start talking about how we can engineer something that's just world-class, second to none. But the fact of the matter is we didn't have to.
Steve Applegate: You know, the minimum security is what you strive for, right? You're not trying to protect anything over what your appetite is or what your understanding of the risk management - you know, what the risk appetite leaves you with. I guess that's back to my first point of governance, right? Because if you really understand governance, you've got a defined risk appetite. You know, you're able to stop without overprotecting and just put in the minimum that's required.
Mark Urban: Steve Applegate, CISO at Dragos. And I'm Mark Urban for this week's Learning Lab. Thanks very much, Steve.
Dave Bittner: And that's "Control Loop," brought to you by the CyberWire and powered by Dragos. For links to all of today's stories, check out our show notes at thecyberwire.com. Sound design for this show is done by Elliott Peltzman, with mixing by Tre Hester. Our senior producer is Jennifer Eiben. Our Dragos producers are Joanne Rasch and Mark Urban. Our executive editor is Peter Kilpe, and I'm Dave Bittner. Thanks for listening.