Gleaning OT insight from the ISACs.
Dave Bittner: It's February 8, 2023, and you're listening to "Control Loop." In today's OT cybersecurity briefing, multiple strains of Russian wiper malware are targeting entities in Ukraine. A high-severity command injection vulnerability affects Cisco devices. The IoT supply chain is threatened by exploitation of Realtek Jungle SDK vulnerabilities. And U.S. Congressman Andrew Garbarino will serve as the new chairman of the Subcommittee on Cybersecurity and Infrastructure Protection. Our interview today is Part 2 from Dragos' Ask the ISACs discussion led by Dawn Cappelli, Dragos' head of OT-CERT, with panelists Tim Chase from the MFG-ISAC, Eugene Kipniss from MS-ISAC, Jennifer Lyn Walker from WaterISAC and Matt Duncan from E-ISAC. In today's Learning Lab, Mark Urban is joined by Dragos' Lesley Carhart, and they discuss Part 2 of creating an ICS- and OT-specific incident response plan.
Dave Bittner: ESET says a new strain of wiper malware they're calling SwiftSlicer has been deployed against Ukrainian networks. The researchers believe the malware is operated by the Sandworm APT, which has been attributed by the U.S. government and others to Russia's GRU. CyberScoop quotes Jean-Ian Boutin, ESET's director of threat research, as saying that the attack focused on a specific target in Ukraine's public sector. The impact of the attack is unclear. ESET also disclosed, in its APT activity report for the third quarter of 2022, that Sandworm used another previously unobserved strain of wiper malware against a company in Ukraine's energy sector. This malware, which ESET tracks as NikoWiper, uses the Windows command line tool SDelete to permanently erase data. The researchers note that this attack occurred in October 2022, around the same time the Russian military began launching missile strikes against Ukrainian energy infrastructure. While it's not clear if the two events are directly related, ESET says it suggests that Sandworm and military forces of Russia have related objectives. Additionally, the Ukrainian Computer Emergency Response Team, on January 26, reported identifying five distinct strains of wiper malware in the networks of the Ukrinform news outlet. The wipers targeted systems running Windows, Linux and FreeBSD. A Russian hacktivist group claimed responsibility for these attacks.
Dave Bittner: Trellix discovered and responsibly disclosed a remote command injection flaw affecting multiple Cisco appliances, including some used in industrial environments. The affected devices include 800 Series industrial ISRs, IC3000 industrial compute gateways and IR510 WPAN industrial routers. The researchers explain that the flaw, tracked as CVE-2023-20076, can be used to gain unrestricted access, allowing malicious code to lurk in the system and persist across reboots and firmware upgrades. Trellix stresses that this vulnerability can also be used to compromise third-party devices in the supply chain. Cisco has issued patches for the devices, and customers are urged to apply them as soon as possible.
Dave Bittner: Looking at attack records between August and October of last year, Palo Alto Networks' Unit 42 researchers discovered that one vulnerability in particular, a remote code execution issue affecting the RealTek Jungle SDK, was particularly attractive to attackers. Unit 42 says it's unusual to see a single vulnerability account for more than 10% of the attacks detected over a period of time. But this one, CVE-2021-35394, accounted for more than 40% of the total number of attacks over those three months. The researchers state that many of the attacks we observed tried to deliver malware to infect vulnerable IoT devices. This tells us that threat groups are using this vulnerability to carry out large-scale attacks on smart devices around the world.
Dave Bittner: Representative Andrew R. Garbarino, a Republican from New York's Second District, has been appointed to chair the U.S. House Subcommittee on Cybersecurity and Infrastructure Protection in the new Congress. Garbarino previously served as a ranking member on the subcommittee. The congressman said in a statement that he looks forward to continuing to work closely with CISA and fostering a strong partnership and open dialogue between the public and private sectors in order to face rising threats and strengthen our national cybersecurity posture.
Dave Bittner: In our interview today, it's Part 2 of Dragos' Ask the ISACs discussion led by Dawn Cappelli, Dragos' head of OT CERT, with panelists Tim Chase from the MFG-ISAC, Eugene Kipniss from the MS-ISAC, Jennifer Lyn Walker from the WaterISAC and Matt Duncan from the E-ISAC.
Dawn Cappelli: So how about you, Jennifer? How about water and wastewater? What's the year been like for your sector?
Jennifer Lyn Walker: Sure. Thanks, Dawn. So I always hate going first, but now I'm - the downside in not going first is that your colleagues say so many things that you want to touch on, too. So, you know, first and foremost, I echo what my colleagues have said from the MS-ISAC, Eugene, and Matt from the Electricity ISAC thus far. Definitely have to echo that, and I'll talk a little bit more about that. But one other thing that - so we serve the U.S., as well as some international water and wastewater utilities. As Eugene mentioned in the beginning, there's - and - as well as Matt, there's also - there's some crossover between sectors. And the one thing that I want to start off by saying is that if, you know, you're a member of an ISAC - and I certainly - you know, we certainly want members for the small and medium utilities. It is a nominal fee for small utilities.
Jennifer Lyn Walker: But if you are a municipality, I highly recommend you - there is no excuse not to belong to the MS-ISAC. And if you can't afford, you know, another ISAC if it's a paid - you can certainly belong to multiple ISACs at a time. But municipalities have absolutely no excuse not to belong to the MS-ISAC and have that, you know, plug into that landscape and what's going on, especially if you don't, you know, have internal staff that can track the threats and whatnot. Again, MS-ISAC is free. So that's my plug for MS.
Jennifer Lyn Walker: That said, you know, we are seeing pretty much the same as everyone else. The bulk of the reports that we're, you know, receiving do involve - are attacks to IT, such as the business email compromises and the ransomware. And besides a few ransomware incidents that are impacting, you know, an HMI or an engineering workstation, we really don't get a lot of the OT-impacting incident reports. Honestly, for our sector especially, I think that's largely because the bulk of our members don't have OT monitoring in place yet, you know, especially the medium - the small and the medium. And so they can't see that threat activity. So we receive reports on what, you know, our members are actually able to see. So to Matt's point about, you know, implementing OT monitoring, you know, of course, we're certainly encouraging that so they can, you know, see the entire threat picture that's impacting their network.
Jennifer Lyn Walker: As far as the - like, the - we talked about, you know, the challenges. You know, the Water Sector Coordinating Council, in 2021, did a survey. We were heavily - WaterISAC was heavily involved in that. And as part of the survey, there was a resulting report called "Cybersecurity - 2021 State of the Sector," where some of the challenges that were identified by those participants of the survey - and this went out way beyond WaterISAC membership. We had some of our sector associations help amplify the survey, and so it went out to a really wide, you know, swath of utilities. And we did have - I can't remember what the percentage is, but we did have quite a number of the smaller ones respond. So that was extremely encouraging that our partners had those reaches.
Jennifer Lyn Walker: But the challenges that they identified were a need for sector-specific cybersecurity training and education, a need for technical assistance with assessments and tools, even how and where to start with those tools. Now, the larger systems have been addressing cybersecurity. When I speak to, you know, when I speak to any audience these days, I'm giving - I'm kind of issuing a charge to the larger and the more cyber-mature utilities to be a good neighbor and help get the word out and invite the - their smaller, rural neighboring utilities, whether it's a electric co-op, you know, or another - a small, rural water - or rural water utility. But invite them to events like this or other cybersecurity events that they're attending or even tabletop exercises that they're going to or that they're hosting - you know, that overall awareness of the threats and the resources to assist, again, mostly for the smaller systems that don't know where to go.
Jennifer Lyn Walker: And they also cited a need for federal loans and grants, which we know is coming to fruition, to help with all of these other things, so that's really good - including the funding to hire cybersecurity personnel, which, again, also deals with the challenges of getting support from executives for IT security, let alone OT cybersecurity. And one other notable need, though, that was identified - and this was largely from the smaller utilities - was that they didn't know what they needed. It was - this was, like, 18% of the respondents. It was - there were 600 respondents to the survey - over 600. So, like, over 100, you know, really indicated that they didn't know what they needed. So I think that's very telling. And while it's not part of - wasn't part of the survey, our members also have a challenge of often being, like I said, integrated into Eugene's constituents with the municipalities and authorities, which can sometimes blur those lines in, you know, responsibility or accountability ownership as well as, you know, funding for their portion, you know, of the - you know, the wastewater treatment of their portion of the municipality - water and wastewater treatment.
Jennifer Lyn Walker: So as far as how we help our members respond, we monitor the threat landscape so that our members don't have to and report back out, you know, the information in, you know, context. We also have, you know, our relationship with our other ISACs and our other federal partners and - as well as private firms such as Dragos. But we can recommend resources and offer guidance. Unfortunately, WaterISAC, we're a small team, and we don't have a full analysis capability. While I have a background in malware analysis and investigation, we don't really have the supporting staff and resources in place to assist in an incident-response capability on that level. But, again, we have Eugene - partnerships with Eugene and Matt and CISA and - as well as Dragos' OT-CERT. So that's one reason why we value these partnerships, to be able to help our members out with that.
Dawn Cappelli: All right. Thanks. OK, now over to Tim. So, Tim, manufacturing is kind of different, at least in my mind. You know, you think power, water.
Tim Chase: Right.
Dawn Cappelli: You think more about, like, nation-state threats, at least in my mind, although we heard from everyone before you that ransomware is an issue for everyone, but...
Tim Chase: Absolutely.
Dawn Cappelli: ...How about manufacturing? I think of that a little differently.
Tim Chase: Yeah. So I think that you made a really good point, which is - and we'll just kind of start off as a level set - is that all the other sectors that have spoken so far are truly industry verticals, right? Manufacturing is not actually an industry vertical so much as it is an industry horizontal. It's a modality that supports every industry, right? And what the threats are to them largely depends on what sector they're supporting. So nation-state activity is a major threat, but usually that is more for a DIB space or something. They want to steal, you know, intellectual property on, you know, the next fighter aircraft or bomber or something else like that. Or we saw a lot of nation-state activity around the biopharma community as we moved into COVID, trying to steal intellectual property related to vaccines that were being developed, right? There's many different other sectors. And I would say, overall, most of the manufacturing is not in that DIB space in the U.S. And while there are some very large manufacturers, even below those manufacturers - right? - it's their giant supply chains. It gets very, very small, very quickly.
Tim Chase: So what is the security and cybersecurity of that supply chain? So the different manufacturers have a different kind of perspective on where they're at. So for Mars, they have a very well-developed program, and we worked with them in partnership with KPMG and several other consumer package group to actually look at third-party risk, supply chain security. So they're in a very high level, right? But then most manufacturing - trying to support them is really about cyber hygiene and helping them understand, as Jennifer was talking about - they want to know what they should do - right? - and how they rack and stack and prioritize some of those things. And so, you know, how we're helping and who we're helping is sort of where they're at in that security maturity model, which is largely dependent on sort of the size of the organization. Occasionally, though, the size of the organization does not necessarily correlate very well to how mature they are. They can sometimes get fairly large and still not really mature that much.
Tim Chase: But on the ransomware, ransomware has been a really negative scourge. I think Matt used the word scourge, and it is. But the only positive thing about it is that for - manufacturing, for many, many years really wasn't focused on cybersecurity. Ransomware is the first time that, you know - and one of the activities that was really kind of getting the manufacturers' attention because there are a number of manufacturers that have just folded. They've had to go under just because of a ransomware incident. So it's helping to kind of get the message out there. And one of our messages to those manufacturers is, you know, cybersecurity is business security. They've got a safety culture, but not necessarily a cybersecurity - so kind of cyber safety and business risk. So that's why, you know, partnerships with, like, OT-CERT and the like are important because there's resources to help, you know, self-assess and, you know, do some tabletop exercises and the like. And so one of the ways that, you know, we're trying to help is - I want to answer Ben's question, too, about state and provincial entities. We're sort of - a manufacturing ISAC's probably not the best way to do that, but in manufacturing, there are MEP networks - stands for manufacturing extension partnerships - and it's Department of Commerce money that helps promote manufacturing in individual states. So there's an MEP, and that's one of the ways where some dollars can go out for some of that training and sort of rightsizing of security programs. So that's one of the ways that kind of, on a state-by-state or provincial level, manufacturers can be help.
Tim Chase: I would also point to a GRF. Or we have a K12 SIX program, which is kindergarten through 12th grade. And one of the ways is that - that we're trying to get around to that is State Board of Education, all these districts - school districts is trying to pool resources, including money, to buy down the cost of certain services, like MSP services, that could be distributed more broadly. So that's one of the ways we're doing it.
Tim Chase: And the second question about, like, cloud-based SaaS - I actually moderated a panel at a conference in Miami, and our panel was OT adoption of cloud. And I think that manufacturing is actually light years ahead of this in - than other industries, mainly because it's less of that truly industrial. And so it's easier and quicker, especially as lines change and, like, products being developed are changing. They can swap those very quickly virtually. So there is a large adoption of that.
Tim Chase: And the pros and cons of that are, you know, that a lot of people are concerned about security around that. I think that our conclusion was that if done properly, the security gains of being able to have one pane that you can actually view so you can actually make security changes at scale and speed outweigh some of the risks. But manufacturing is different. And relating to individual members largely depends on how large they are and where they are in that security maturity model.
Dave Bittner: In today's Learning Lab, Mark Urban is joined by Dragos' Lesley Carhart, and they discuss Part 2 of creating an ICS/OT-specific incident response plan.
Mark Urban: Hello once again, Mark Urban with the Learning Lab on "Control Loop." Today's topic is incident response for cyber incidents in industrial control systems and operational technology. And to help us understand that world better, I'm joined by Lesley Carhart, one of our own here at Dragos, the director of Incident Response for North America. Welcome, Lesley.
Lesley Carhart: Thank you for having me, Mark.
Mark Urban: So we've got key elements of the relationship, the environmental understanding, the cyber functions, the monitoring, vulnerability management, looking at the consequences. Are there more? What are any other elements of that sort of response plan out there?
Lesley Carhart: That's going to be a big start to your plan. If you don't have those fundamental pieces of understanding about what does bad look like; what do we have to secure; who is involved in securing it; what do we have as challenges in this environment; it's going to be hard to do anything. So I don't want to overwhelm people with a million different pieces that you could potentially have in your incident response plan because you can get quite detailed over time as you build your maturity.
Lesley Carhart: Start with those basics. Start with those basics. Start there, and start building your plan around those fundamental things. What are you worried about? What's out there to secure? And also make note of who is going to do which thing. Who are you going to call? What if it's 2 a.m. on Christmas? Because cybersecurity incidents always happen at 2 a.m. on Christmas when everybody's at - with their families and potentially drinking. That's when you have - the adversaries know when to attack. They know when you're out with your families, and nobody's watching things, and nobody's ready to respond. So you need to have a plan for who and what and when if you have something negative happen.
Lesley Carhart: You're going to have an idea of priorities once you start doing this consequence modeling and you understand your architecture and what's vulnerable more. You'll have a better idea of risk assessment. And that's - in the end, risk is going to be a decision made often by the plant facility, not by you as a cybersecurity professional. That's related to direct health and safety risk. Your plant manager is normally going to make that type of decision. But you do need to know when you're going to activate your incident response plan and who is going to be called.
Lesley Carhart: And what communication method are you going to use? What if it is a holiday? What if it is a weekend? How are you going to reach the right people? What if they don't pick up their phone? What do you do then? So you need to start planning for that. Who's the most essential? Even if you're using, say, Dragos, as an incident response retainer provider, what's the phone number? Who's going to activate your contract? Who's going to be the decision-making authority on that? How do you call Dragos? We provide you some of that information when we set up the retainer, but you need to have it in a central location people can access, even if it is 2 a.m. on a holiday - so having that essential plan.
Lesley Carhart: And the great thing about starting to write down that plan is it can be fluid. You can change it in the future. It's OK if you get some things wrong, but you can test it really easily, right? You can sit down at a table, at your lunch table in your office, and you can walk through, OK, we've just had this happen, let's talk through how we'd activate this plan we just wrote. And you could work out the kinks that way. And it's low pressure. Nothing's going wrong. Again, plan this in advance. Don't do it when you have a crisis. Anybody can have a crisis, and you need to be generally prepared for what you're going to do. So take it easy. Take the time now when things aren't on fire, and start building that plan of the who, what, wheres, whens, and try it out.
Mark Urban: That makes sense. And I, you know, appreciate not only overwhelming - not overwhelming, rather - giving a couple of good operable sections to look after and not overwhelming. But what are some of the issues? So I imagine when you go into environments and you're creating this plan - is that an easy thing to do? Do you ever see, you know, issues in the creation process of running through that plan?
Lesley Carhart: Sure. People try to get ahead of themselves, and that's usually the No. 1 problem in building all cybersecurity is getting ahead of yourself. In anything, whether that's building security monitoring, building a cybersecurity plan, building an incident response plan, installing cybersecurity tools, there's a lot of sales pressure out there. There is a lot of things going on in the larger cybersecurity world that can seem overwhelming and scary. And that means that sometimes the very basic things are missed. And I've talked about some of those, like just understanding what's in your environment and what you're worried about and what people are involved. Sometimes people jump ahead to really advanced technologies that they don't understand how to deploy yet, they can't support, instead of just working on those fundamentals of architecture reviews and having a basic incident response plan in place and maybe getting an incident response retainer and getting basic, fundamental monitoring in place. Those things should be ahead of all the most advanced planning and the most advanced technologies.
Lesley Carhart: We need to get the basics done first, and that's usually the biggest problem in pretty much all areas of cybersecurity but certainly in building incident response plans. Start with those fundamentals. Test them out. See if they work - your activation plan, how you identify an incident, who gets called. Try your call trees. Make sure you can reach somebody even if they're not in the office - fundamental, basic stuff. But knock that out first. If you try to do too much at once, you're going to get overwhelmed. It's going to be too daunting. Start with the basics.
Mark Urban: Keep it basic. Do the fundamentals. You know, you'll come to the point, maybe down the road, where you can get fancier and more advanced technologies. Blocking and tackling is the best way to start. OK. Good. OK.
Lesley Carhart: It'll grow with you over time.
Mark Urban: Gotcha. And what - you know, so as you, you know, you respond to incidents, what do you see typically goes wrong with plans or might go wrong with an organization when, you know, when the heat is on, when, you know, when the worst day happens? What can go wrong?
Lesley Carhart: Really, the most common thing that I see is lack of preparation that leads to a lot of extra expenses. And, again, I'm not trying to be a salesperson here at all. Really, this is just facts. What I see happening is organizations don't have a good incident response plan, and they don't think they're going to be a target, and then their really bad day happens. They aren't sure whether it's cybersecurity related, or they have a ransomware incident, and all their computers are locked, and they don't have a plan. They don't have a plan for when to activate. It takes them too long. They don't reach the right people, and they don't have a adequately prepared incident response team. So they go out on the internet. They try to find somebody to do incident response for them.
Lesley Carhart: And the unfortunate thing in the market right now is that there is a huge demand for incident response, and the companies that are legitimate and trustworthy who offer incident response usually do things on a retainer basis with best effort for ad hoc calls. And this is kind of universal across the enterprise space and the ICS space. And if you call at the last minute and you haven't put anything in place in advance, they might tell you, no, you have to wait a few weeks to get response because you don't have a retainer. You haven't planned anything with us in advance. You're not an existing customer, and we prioritize those.
Lesley Carhart: And so what happens is these organizations will call the first - they'll go down the line on Google, and the first person who offers to sell them incident response is who they'll go with. And the unfortunate truth of the world is not every organization who's aware of the need for incident response and is willing to sell it to you is really well-qualified. And certainly, not all of them know how to do industrial incident response. So I go into a lot of environments where I end up having to redo entire incident response efforts, and those are at pretty substantial hourly rates. There's no retainer in place. And this organization that didn't understand industrial technology or wasn't very good at incident response got called in, and they did the investigation wrong, and they didn't get a good root cause, or maybe they damaged some equipment themselves. And so I have to start from scratch and try to redo the entire incident response effort and give them better advice.
Lesley Carhart: And, again, I'm not trying to fearmonger. I'm not trying to sell anything. But - and certainly not saying we're the only organization out there that can do incident response in industrial environments. But the important thing is to have that plan, know who you're going to call, whether that's internal or external or a hybrid. Have those basic pieces of documentation about your architecture and your asset inventory. You don't want to pay me or another hourly rate consultant at the last minute, ad hoc to do basic asset inventory for you. I feel horrible when I have to do that for an organization. It's very expensive. It takes extra time that could be used for investigation. I'm doing those fundamentals at a much higher rate than their employees because they were never done, and they have to be to do good incident response. So whoever you get, whether it's internal or external, to do incident response, make sure that your documentation is done, and they're prepared, and you've tried things out, and you've gotten these basics done before jumping ahead. It's going to save you a lot of money and a lot of heartache in the long run.
Mark Urban: OK. That's good guidance. Let's assume that somebody has taken your good guidance. Let's assume they have a good plan in place. How would you describe, from your perspective, when they encounter an incident, when they have to activate that plan? How do you characterize those moments that - when that happens?
Lesley Carhart: Oh, gosh, Mark, the first day of incident response is always really scary. It doesn't matter how prepared you all are. It's always going to be scary. It's always going to be stressful. You're doing crisis management. You're doing disaster recovery and with added pressure because of these health, life safety concerns in an industrial environment. However, you will feel so much better if you've drilled the plan, you've tried it out. There is documents and cut sheets you can refer to, like an incident response plan and its associated playbooks. It's still going to be stressful. It's still going to be a really bad day. You've had an intrusion. Yeah, that sucks. It sucks for everybody. It's never fun.
Lesley Carhart: And, again, anybody can be in a target, and anybody can be intruded upon. It can happen to anybody of any size or any preparation. And it's always icky. It's always stressful, and something's always going to go wrong, too. But the more you plan, the more you prepare, the more you drill, drill, drill, just like any safety plan, the better it's going to go and the more effective, the more efficient it will be and the less stressed out you will be in the long run, the less potential for error there will be. You'll miss less things because you're going to be following a plan. It's going to be written down. You'll have done it before. It's not going to be the same when you're actually dealing with a crisis instead of just a drill, but you're going to do better. That's the important thing.
Mark Urban: OK. Well, that certainly puts the impetus to get it done ahead of time because even though it's a terrible time, you at least have that plan to fall back on or as a baseline to operate within, even if, you know, alarms are sounding, and the pressure is on because of all those - you know, because of all those characteristics that you named. But outside of the documentation - and we're kind of drawing to the close - outside of the documentation, outside of the plan, are there any characteristics that are important, you know, to do it right in that moment that you've seen in your encounters?
Lesley Carhart: Don't panic. That's the No. 1 rule for incident response. And it's going to be very tempting to panic because, especially with this industrial equipment, you're talking about lives on the line. Yeah, that's scary. But, you know, there's other professions out there that deal with crises. And I think if you talk to most people who do firefighting or medical first response, work in emergency rooms, they'll tell you the same thing. It's don't panic. Panic doesn't solve anything. Try to stay as calm as you can. The adrenaline will be going. You'll be stressed out. It'll be a bad day. You'll be tired. You probably won't get enough rest. But don't panic. That's not going to accomplish anything. Try to calm down and follow the plan. Follow the plan. Follow what you've practiced. It's OK to be stressed out. It's OK to be nervous. It's OK to have the adrenaline running. It's OK to make a mistake, even. You're going to make mistakes. It's going to happen. But the more you plan, the less likely those are to be impactful, dangerous mistakes.
Mark Urban: Lesley Carhart, director of Incident Response in North America for Dragos, thank you so much for your time today and giving us that insight into not only what happens on that day, but how to - for those worst days, how to prepare for them with a plan. Thank you, Lesley.
Dave Bittner: And that's "Control Loop" brought to you by the CyberWire and powered by Dragos. For links to all of today's stories, check out our show notes at thecyberwire.com. Sound design for this show is done by Elliott Peltzman, with mixing by Tre Hester. Our senior producer is Jennifer Eiben. Our Dragos producers are Joanne Rasch and Mark Urban. Our executive editor is Peter Kilpe, and I'm Dave Bittner. Thanks for listening. We'll see you back here next time.
