Compliance with an eye on OT security.

[ Music ]

Dave Bittner: It's July 26th, 2023 and you're listening to Control Loop. In today's OT cybersecurity briefing, an unnamed APT has a remote code execution exploit for Rockwell Automation ControlLogix communications modules. A court temporarily blocks water system cybersecurity mandates. Industrial controller vulnerabilities pose a risk to critical infrastructure. The White House publishes an implementation plan for the international cybersecurity strategy. The US Federal government issues voluntary IoT security guidelines. Our guest, Mea Clift of Woodard & Curran, talks compliance with an eye on OT security The Learning Lab concludes with the final part of a three part discussion between Dragos' Mark Urban and Vulnerability Analyst, Logan Carpenter. They are talking about vulnerabilities in the OT world.

[ Music ]

An unnamed APT is in possession of a remote code execution exploit, affecting Rockwell Automation ControlLogix's communications modules, BleepingComputer reports. Rockwell has issued patches for all affected products, and organizations are strongly advised to apply them. Rockwell analyzed the vulnerability with assistance from the US Cybersecurity and Infrastructure Security Agency. And the company believes there's a high likelihood that these capabilities were developed with an intent to target critical infrastructure, and that victims' scope could include international customers. Dragos said in an analysis of the vulnerability, "Knowing about an APT owned vulnerability before exploitation is a rare opportunity for proactive defense for critical industrial sectors." The type of access provided by CVE-2023-3595 is similar to the zero day implied by Xenotime in the TRISIS attack. Both allow for arbitrary firmware memory manipulation, although CVE-2023-3539 targets a communication module responsible for handling network commands. However, their impact is the same. Additionally, in both cases, there exists the potential to corrupt the information used for incidence response and recovery. The US Court of Appeals for the 8th Circuit has granted a temporary stay of an EPA memorandum that would require states to evaluate the cybersecurity of their water systems, The Washington Post reports. Agency spokesperson Robert Daguillard told The Post, "EPA is disappointed by the Eighth Circuit Court of Appeals' order that undercuts EPA's efforts to protect the safety of the nation's drinking water from malicious cyberattacks." The one-sentence ruling offered no reasons for the temporary stay, simply stating, "The motion for stay of the Environmental Protection Agency's March 3rd, 2023 memorandum pending disposition of the petition per review is granted." Three states' attorneys general petitioned for the stay, and they did so with the support of several water utility associations. The petitioners' public statements have emphasized their skepticism over the EPA's proposed rules, which they regard as representing a simplistic one-size-fits-all approach to water systems' cybersecurity. They also objected to what they characterized as a heavy financial burden the rules would impose on smaller utilities. The EPA, for its part, has emphasized the troubling frequency of cyber attacks against water systems. The utilities do rely heavily on network operational control systems for their routine operations. A recent example of a cyber threat to a water system is the one that affected the Discovery Bay Water Treatment Plant in California. That attack drew a federal indictment. Researchers at Armis discovered nine vulnerabilities affecting Honeywell's Experian Distributed Control Systems products, TechCrunch reports. An attacker with network access could exploit the flaws to remotely run unauthorized code on both the Honeywell server and controllers. Curtis Simpson, CISO at Armis, told TechCrunch, "Worst-case scenarios you can think of from a business perspective are complete outages and a lack of availability. But there's worse scenarios than that, including safety issues that can impact human lives." Honeywell issued patches for the flaws last month. Honeywell spokesperson Caitlin E. Leopold said in a comment to TechCrunch, "We have been working with Armis on this issue as part of a responsible disclosure process. We have released patches to resolve the vulnerability and notified impacted customers. There are no known exploits of this vulnerability at this time. Experion C300 owners should continue to isolate and monitor their process control network and apply available patches as soon as possible." The National Cybersecurity Strategy Implementation Plan the White House issued earlier this month has five pillars. All of them are of interest to operational technology and industrial control system operators. But the first pillar, defending critical infrastructure, has particular relevance. That pillar has five strategic objectives that are, in turn, supported by specific initiatives. The first strategic objective established cybersecurity requirements to support national security and public safety is self explanatory. And the objectives suggest that the government believes current regulatory regimes are inadequate to the task. The second strategic objective, scale public-private collaboration, tasks the sector risk management agencies responsible for each of these 16 critical infrastructure sectors with developing secure by design and secure by default principles that would advance their sector security. Integrate federal cybersecurity centers is the third objective. The single initiative here mandates a review to identify capability gaps. The next objective, update federal incident response plans and processes, aims at developing such plans and processes into a comprehensive hole of nation approach to cyber incidents. It seeks to make response quicker, more immediately responsive to warnings, and to develop training that will enable the responders to work effectively. Tabletop exercises are particularly called out. The fifth strategic objective, modernize federal defenses, concentrates mostly on IT systems with special attention paid to federal civilian executive branch agencies' systems. The White House points out that the guidance is not exhaustive. Agencies are expected to take actions appropriate to their missions and circumstances. Operators of critical infrastructure might begin by getting close to their sector risk management agency, their SRMA. Since public-private partnership is called out repeatedly in the implementation plan, companies would do well to take the document at its word and not hesitate to reach out to the appropriate federal offices. The White House has announced a cybersecurity labeling programming for smart devices. Under the proposed new program, consumers would see a newly created US cyber trust mark in the form of a distinct shield logo applied to products meeting an established cybersecurity criteria. The goal of the program is to provide tools for consumers to make informed decisions about the relative security of products they choose to bring into their homes. Manufacturers and retailors that have committed to the voluntary program include Amazon, Best Buy, Google, LG Electronics, Logitech, and Samsung. According to CyberScoop, the program will be overseen by the Federal Communications Commission. The Washington Post earlier last week interviewed FCC Chair Jessica Rosenworcel. The choice of the FCC as the responsible agency emphasizes that connected devices, and realistically, that means at some point wireless connectivity, will be the devices that will qualify for or fail to qualify for the badge. Rosenworcel told The Post, "We live in an era of always on connectivity. Connections aren't just convenient, they power every aspect of modern life. And if this energy is new, I would say our authority is old. We're just giving it modern meaning. And I think in a modern way, that requires us to think about how to make communications networks cyber secure." She also offered some thoughts on network security, stating, "We have issued a list of equipment that we believe is insecure that we won't support on our networks." Here, she's clearly referring to the Rip and Replace program that addresses concerns about the security of Chinese manufactured hardware. She also wants to see her commission continue to work to understand the vulnerabilities in the border gateway protocol. The whole effort, Rosenworcel believes, is inherently an interagency one. Whichever organization takes the lead. She told The Post, "I don't think this task is one where the agency succeeds on its own." She hopes to increase coordination with other agencies from across the government. Different agencies have different missions, different histories, and different equities, which should enable them to make distinctive contributions to the common task of securing connected devices. What sorts of devices might be up for a cyber trust mark? Rosenworcel mentioned connected refrigerators, microwaves, televisions, climate control systems, fitness trackers, and baby monitors. The Post points out several potential gaps. Speakers, doorbells, security cameras, and cars. But after all, the list Rosenworcel reeled off was an informal one. The cyber trust mark is intended to be a caret, and not the sort of stick one often associates with regulatory action. Rosenworcel thinks the labels might begin to appear by the end of next year at the end of 2024. "These things don't move fast," she said. And cautioned that her prediction wasn't a commitment to a timeline.

[ Music ]

I recently had the pleasure of speaking with Mea Clift, Head of Cybersecurity for Woodard & Curran. Our conversation centers on compliance, with an eye on OT security. Here's my conversation with Mea Clift.

Mea Clift: I think that water and wastewater is still learning how to deal with compliance in an ever changing landscape. We have new regulations coming down. We see water is listed in the government mandates as critical infrastructure. So that changes the perspective of these small organizations that may not have thought about cybersecurity in that way before. They've looked at physical security, they have gates and they have locks and doors. But they haven't had an opportunity to really look at understanding OT environments need cybersecurity just as much as their laptops on the regular network. So they're still really learning that those policies that they've built for the IT network aren't really in concert with how OT works. And so they have to learn how to marry them, but they also have to learn how to compromise and lean into the differences that OT has compared to IT.

Dave Bittner: For folks who aren't in that water world, can you give us a little idea of how it's set up? I mean, I think most -- electricity seems to get all of the attention when it comes to press and understanding how it works. But not a whole lot of people I think who aren't in that world really get how the water utility space functions.

Mea Clift: It really depends on the organization. So you have, you know, drinking water. And you have wastewater. So there's two different types of water plant, I think there's more than that, but we'll focus on those two. So drinking water is keeping that water clean, putting the right chemicals in it. And then how to pump it out, and then that's where we see water main breaks, that's where we see chemical boil advisories because of contamination or there was even a case in Vermont - I think it was Vermont - where a water manager didn't put fluoride in the water for like 10 years.

Dave Bittner: Wow.

Mea Clift: Because he didn't believe in it. So that's kind of the drinking water space. Then the wastewater space is, you know, where does everything go when it leaves your sink, when it leaves your bathroom? And how do we treat that to get that water back into usage? Because we only have so much water in the world, right? So we have to do what we can to, you know, take solids out, to clean those things, to make water good again. So there's a lot that goes into that. And those operations have to be able to run 24/7. Because I don't know about you, but I really like to drink water. And I like having showers. So, they really are critical and up until recently, they didn't have to worry too much about cybersecurity because they're running on older software. Or sometimes they may not be running on software at all. They may be running manually. But with the advent of the pandemic and people putting in remote solutions and starting to identify that there were ways that people could do some of their work on these plants remotely, it did open up the industry for exploit and put, you know, added them to the threat landscape.

Dave Bittner: So when we're talking about compliance. So can you give us an idea of what the -- what's the spectrum of regimes here? Are we talking about a federal level, a state level, a local level? Do they all come into play?

Mea Clift: The America's Water Infrastructure Act of 2018 really started the ball running on understanding compliance around the OT space. And it requires that every water plant has to do an inspection from a cybersecurity perspective and do a risk assessment every five years. And in that, they're supposed to have things like incident response plans, emergency response plans. And both of those should have cyber components in them. Along with making sure that they know what assets are on their network and how things are programmed effectively. And building on that maturity. It doesn't say you have to do this or you'll be fined, it's just saying this is what we recommend. The EPA this year then started with the mandate of having sanitary inspectors in every state do cybersecurity hygiene inspections as well. But that of course was just recently blocked. So in the future, federal mandates are likely going to be what perpetuates good cyber hygiene in the space. But we're still waiting to see what that's going to mean.

Dave Bittner: What is the state of things when it comes to cyber hygiene? Are folks -- like a lot of utilities out there, my sense is they're playing a bit of catch-up?

Mea Clift: They're very much playing catch-up. And I think a lot of times, there's still a pervasive mentality of that won't happen here. Or we're totally okay. We don't have things, or our IT department's going to take care of it. And in reality, the IT and OT environments need to be completely separate. We've seen a lot of ransomware attacks take out water systems for municipalities and having to make them run manually instead of running with technology because the IT and OT networks weren't separated and someone clicked an email and installed something that took over the network. Isolating the OT network, because the technology being older than modern constructs, allows it to be insulated and adds an extra layer of security that it needs.

Dave Bittner: Yeah, that's a really interesting insight. That, you know, I guess there's an attitude of if it ain't broke, don't fix it when it comes to a lot of these things. I don't know, perhaps I'm being naive. But a valve can function for decades.

Mea Clift: It's true, but there's also the challenge of budgeting. You know, you have these small municipalities who have to decide on whether securing their remote software to their water system is affordable when they have three water mains that are 100 years old that are cracking and need to be replaced before you lose water and have boil water advisories and that mess with the reputation or even access to the drinking water that their community needs. So, I think with some federal intervention with grants and enhancements in cybersecurity, you know, CISA's done a lot, there's a lot of information sharing organizations like Water ISAC. I know of course Dragos has ITServ. I think that there's progress that can be made. It's just a matter of raising the awareness and then helping to build that availability to these environments to give them the resources needed to balance out paying for that water main and paying for cybersecurity.

Dave Bittner: So, what are your recommendations then for organizations to approach compliance? What's a practical way to come at it?

Mea Clift: Start with the foundations. Look at your network, split them up, you know, separate your IT and OT as much as you can. Hopefully completely physically separated. And then also, look at incident response, look at how you're protecting things. Even if it's as simple as don't write down your password and leave it on the HMI system, you would be amazed at how much of a difference that can make. Because people just think it's okay to leave their password out. And that's not okay anymore. And then also, you know, working on educating not just about their OT network and how it affects the water, but looking at things like fishing and ransomware. Raising those awareness components I think is really important. Then you can work on the higher maturity things of do we need to back up these systems, how do we keep resilience, how can we better protect our remote access solution with multi-factor authentication, do we need to upgrade, you know, our controls because they're legacy? Work on those capital improvements after you get the basics down.

Dave Bittner: Do you have a certain amount of empathy for folks who are out there who are saying, you know, we're fighting a bit of fatigue here? As you say, we've got limited budget, limited personnel, limited resources to be able to simply check these boxes sometimes. I mean, we're doing the best we can here.

Mea Clift: I absolutely have empathy. It's a hard position to be in when you have to meet the needs of the people but also be secure to meet the needs of the people. And I think there is a lot of fatigue. Because there's a lot going on. There's a lot of threats, there's a lot of conflicting priorities. And that's why I'm hopeful that in the future, we'll see the federal government provide more grants and available resources. And hopefully other industry professionals like, you know, like how Dragos is doing ITServ and how Water ISAC is building partnerships, will have more opportunity for more public-private collaboration to get the tools and resources to the masses effectively.

Dave Bittner: Our thanks to Mea Clift from Woodard & Curran for joining us.

[ Music ]

Dragos' Mark Urban and Vulnerability Analyst Logan Carpenter finish up their three part discussion about vulnerabilities in the OT world.

[ Music ]

Mark Urban: Hi, this is Mark Urban with another edition of "Learning Lab" here on Control Loop. Today I'm joined by Logan Carpenter. And Logan is a Vulnerability Analyst here on the Dragos World View Intelligence Team. There's a long reporting train that goes into working with the vendors, doing the disclosures. Can you describe when, you know, the term zero day vulnerability is thrown around. Do you look for those vulnerabilities? What are those, and you know, are they particularly concerning or what's your perspective on that?

Logan Carpenter: So, a zero day vulnerability is a new vulnerability. There's nothing about a particular vulnerability that, a zero day vulnerability that makes it special other than the fact is that it's unreported and unknown to the public. Right? So, right? Like for instance, right? I'm looking at a few different, you know, OT softwares and devices and right now. And I've had vulnerabilities that I have not yet reported to the vendors because I'm still, you know, my assessment is not over yet. Right? And I haven't got all my ducks in a row to reach out to them and go hey, you know, let's go through this process. So technically right now, I have zero days, right? These are vulnerabilities that nobody knows about, that the public doesn't know about. So if they are leveraged, they will work. Because there's no patch, no nothing. It's unknown. Nobody's expecting it. So, sometimes I think people kind of get confused when the hear the buzzword zero days and they think these are like specific vulnerabilities with like specific capabilities, but no. It could be -- a zero day could suck. It could be a dumb vulnerability. Right? It could be a vulnerability that if you executed, it will print "hello" in the corner of your screen, and that's all it can do, right? Like it could be something silly like that, right? Or it could be something more nefarious, you know, like remote code execution. Yeah, the whole term of zero day just means it's a vulnerability that hasn't been disclosed to the public yet.

Mark Urban: Got you. So it's not that, it could be a big, bad boogeyman, but that don't get wound up over the terminology of zero day.

Logan Carpenter: Yeah, definitely don't get wound up. There's a lot of hype in the security industry in general. So, that's kind of one of the parts that we play too on our intel team is we try not to overreact to things and like calm our customers down. Even the public, too. Like because sometimes things just get blown out of proportion. And you know, news outlets, you know, they sell on emotion. So like something hits the news and it's like oh, it's this big deal! It's going to cause this next big thing! And it's like no, actually, it's not that big of a deal. You know, you don't have to worry about it, so.

Mark Urban: What are some of the common types of vulnerabilities you do see in OT? Is it a lot of writing "hello" in the upper right screen? Or what do you see?

Logan Carpenter: So, one of the things that kept me in OT security and like drew me in early in my career was it was so easy to -- like IT vulnerabilities and like, like even like pen testing IT systems and devices, it's so much more difficult than OT systems. Because a lot of the basic like don't do this is done in OT products. Right? For instance, like, a basic vulnerability is like a, you know, what would be known as like cross-eyed scripting, right? So, you have like a web server. And you inject some, you know, text into it. And pretty much like you have a web server, right? You'll send a request but you'll change some of the text in the parameter. And it'll execute some code on the server that the server doesn't know about. Or pull data from the server that the server doesn't know about. Just basic cross-eyed scripting. Those vulnerabilities are almost in every single OT, you know, product that we see. Not every single, but a lot. A lot. I looked at a device last year that was a, it was an industrial router. And it had a vulnerability that, you know, the routers they have like a trace root function that you can go out to, you know, do whatever debugging you need to do in your network. Well the trace root function took the text that the user put in and it just slapped it into a bash script. So you could just cancel out with a semicolon and put whatever bash command you wanted and the server had root privileges. So, those kind of vulnerabilities are super popular. Over privileging. Like run times and run times just like, for a PLC, it's like this system that controls the logic and configuration of the device. It's like the system within the operating system. So like, sometimes they'll over privilege run times. Or the web servers that run on these -- some of these devices will be over privileged. And have root access. So if you do have a vulnerability in that code, you can literally do whatever you want. A popular one now is like the UNC path injection stuff. Lots of undocumented commands. Undocumented commands are like -- are ubiquitous in OT and ICS security. Well, OT and ICS devices. So, what undocumented commands are, you'll have these protocols, or you'll have like web servers as well that have almost like these back doors that vendors use, usually for like maintenance. So if you have like a relay from a vendor and it breaks and the relay, and the vendor has a warranty on it, they want to try and fix it for you, they have like these commands that they don't document, they don't tell, but they're there. And analysts like myself can go and look at the firmware or use like fuzzing techniques to find these commands. And you know, for instance, a good example is like one of our analysts, Sam, published a blog about these guys who, this guy that was writing password crackers for PLCs. And the password cracking software he was selling also was embedded with malware. But that's a whole different topic. But the point of it was the way that he was cracking these passwords of all these PLCs was he had reverse engineered and figured out all the undocumented commands that allowed him to overwrite the passwords. So they were like commands that would allow him to read and write physical memory. And I've seen vulnerabilities like that from literally every single big vendor has done this at least once. They all do it, so. Like the PIPEDREAM malware. BADOMEN. Which affected like some of the OMRON tools. There was a -- one of the commands that they used, that BADOMEN used to enable this backdoor was a command that was not documented in any of OMRON's documentation. You couldn't find it even if you reverse engineered the Sysmac system software. It was not used by Sysmac Studio, which is like the engineering workstation software used to program those devices. It was not in there anywhere. It was a vendor specific command that was designed to do maintenance on the actual file system on the PLC. And they found this. So they were able to do things like activate telnet and send files to the actual root file system. Those are some other, you know, undocumented commands are a real issue as well. So really, the embarrassing ones, like when you think of vulnerabilities, whatever the embarrassing ones is, like the basic security things, these are some of the things that we're struggling with in OT and ICS security now. And like we find these all the time. Zip Slip is another one. Or what is it? Directory traversal? That's the technical term for it. So for those of you who don't know what a Zip Slip is, pretty much it's a feature that was in there intentionally but nobody really knew about it, that if you pack up, if you compress any kind of compressed file, zip, tar, gz, whatever, if you compress it in a way that uses what you call traversal sequences, so if you know Linux, it's like dot-dot-slash, dot-dot-slash, dot-dot-slash to kind of go back in directories? You can literally put a file somewhere else in the file system. Just by opening up the zip file. And it was made popular by like Sync or Sync Security, like in 2018 or something like that. Anyways, a lot of OT systems, especially project files, some of the configuration that they upload and download to PLCS, to like, so like logic files and stuff like that, are generally compressed before they are sent. Or like when a project file's saved, it's compressed using these. And a lot of these systems are vulnerable to Zip Slip, so you can like repack things up, if you have access to like EWS, the engineering workstation, you can drop this special zip folder on there and they open it up and it'll maybe rewrite a library or. Yeah, so Zip Slip is another fun one. That's the fun one that we find. It's like low-hanging stuff. I've seen -- actually, I've seen a post on Mastodon. Somebody actually had posted it in one of our Slack channels. And it was like a, it says something pretty much like people say finding vulnerabilities are hard. And it was like just stop looking for the hard ones.

Mark Urban: There are plenty of easy ones.

Logan Carpenter: Yeah, there's plenty of easy ones. Like you're looking for those buffer overflows when you're sending specially crafted, you know, data that, you know, you don't know. Just the stupid thing has Zip Slip. A Zip Slip vulnerability. Just throw a special zip file on there and it'll, you know, overwrite one of the DLLs in the file system. And then you have access to it. It's like, it's way more simple.

Mark Urban: You know, Danielle on my team said you make vulnerabilities fun. And I think she was actually right. That was the best conversation I ever had about vulnerabilities, I don't know about you. But Logan Carpenter, our Vulnerability Analyst here at Dragos, thank you. Thanks for joining. Thanks for all the kind of cool information about vulnerabilities. And I'm glad we have you out there looking for them with the other folks here at Dragos and kind of giving that context to, you know, how people can manage through. I much appreciate it.

Logan Carpenter: Yeah, thanks for having me. I always enjoy nerding out over vulnerabilities. So whenever you need me to talk about something, I'm always available.

[ Music ]

Dave Bittner: And that's Control Loop, brought to you by the CyberWire and powered by Dragos. For links to all of today's stories, check out our show notes at thecyberwire.com. Sound design for this show is done by Elliot Peltzman, with mixing by Tre Hester. Our senior producer is Jennifer Eiben, our Dragos producers are Joanne Rausch and Mark Urban. Our script is by Tim Nodar. Our executive editor is Peter Kilpe, and I'm Dave Bittner. Thanks for listening, we'll see you back here next time.

[ Music ]