Knocking down the legs of the industrial security triad.

Dave Bittner: Hello, everyone, and welcome to the CyberWire's "Research Saturday." I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down the threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Pascal Ackerman: Pretty much industrial control systems are all around us, right? Everything we touch is either directly controlled by one of these systems or it's fabricated by one of these systems. So an industrial control system is a bunch of peripherals that combine to make a product or ship a good or a service to somebody. 

Dave Bittner: That's Pascal Ackerman. He's a senior security consultant for Operational Technology and threat and attack simulation at GuidePoint Security. The research is titled "GuidePoint Security researcher discovers vulnerability in the integrity of common HMI client-server protocol." 

Pascal Ackerman: Think of it as pretty much the engine in your car that drives everything, that makes sure that your battery gets charged, makes sure that there's hydraulic pressure for your pressured steering column and stuff like that. So these systems are controlling the - well, the process. They're industrial control systems that take care of a certain process. So there's variables like temperature, pressure that go into it. And these systems will monitor these control points, and they will make actions to keep them in a certain acceptable parameters. 

Dave Bittner: And so what is the peril here that you all are exploring? 

Pascal Ackerman: So as I said, these systems control a key value like pressure or temperature. And oftentimes they do that very well themselves. Like, you set them, you set (ph) - for example, I want the temperature to be at 100 degrees Fahrenheit, or if you go back to their resemblance to a car, I want my cruise control to stay at 60 miles an hour. So it will use sensors to look at your temperature or your speed in the car reference. And it will throttle steam pressure or it will throttle your gas pedal in the car to stay within that value. Oftentimes there needs to be some sort of an oversight to make sure that the process is within tolerable maximum or minimum values. And that's done with what we call an HMI, or human-machine interface. So an operator will have one of these HMIs sitting in the control room or wherever he is looking at these values, and he can keep an eye on the temperature. Sometimes there's some trending on there as well that can show you outliers to see over time what your pressure or your temperature is doing. I've always been fascinated with the communication from the process to that HMI and how that works, so I started looking at it deeper and deeper, and that's how I discovered these vulnerabilities. 

Dave Bittner: Well, let's walk through it together here. Can you take us through this journey? What exactly did you discover? 

Pascal Ackerman: So as I was doing an assessment of a large manufacturer that was doing a lot of recipe-based manufacturing - this was food and beverage - and then they send recipes from a controller - the PLC or the programmable logical controller, which is the brain of your automation system - through an HMI, and we were taking Wireshark packet captures. So we're basically sniffing the network, looking at all the traffic, all of the data, traversing the network. And I was seeing these recipe names flowing by in clear text. And that set off my curiosity bell, right? And I started looking deeper into it, and I noticed that they were tied to a particular protocol, and this was for an Allen-Bradley facility. So it was what they called a live data protocol, where they basically have an HMI system, with the system being a server and a client - they were communicating these recipes to each other over clear text, and that sparked the whole research project. 

Dave Bittner: Well, let's continue down the path then. I mean, your curiosity is piqued. Where did you go from there? 

Pascal Ackerman: I started looking at the Wireshark packet captures and started to push it through a Python framework called Scapy, which is really, really handy for dealing with packet captures and actually also manipulating it, some of the data, which we will get to in a minute. So I started filtering out certain values. I was looking at, OK, what - when I see these clear text recipes come by, what else is on that packet? Is there anything that stands out? And I noticed that there was a certain part of the protocol always showed these clear text protocols and the declared tag strings passing by. So I started looking deeper at that, and I started honing down, started filtering out that particular part of the protocol. And I noticed that pretty much any data between the server and the client in one of these HMI systems is sent over clear text. 

Dave Bittner: Now, is this data that's reporting back a status? Is this data that is used to control devices, or is it both? 

Pascal Ackerman: It's both. So one way - like I said, the operator can look at his process and look at temperatures, look at pressures. And then if he notices that something is out of whack, he can go in and change a set point. He can stop the process. He can do anything from that HMI, which is also, then, sent back from the client to the server in clear text. 

Dave Bittner: Now, I've seen enough movies where things go horribly wrong, you know, like "Jurassic Park" or something like that... 

Pascal Ackerman: (Laughter). 

Dave Bittner: ...You know, where there's somebody sitting at a control panel and something starts to indicate incorrectly and they say, oh, we must have a bad sensor, you know, and I'll go check it out. And then they go check it out, and they get eaten by a dinosaur. 

(LAUGHTER) 

Dave Bittner: So is that where we're headed here, that the reliability of the information flowing back-and-forth is really at issue? 

Pascal Ackerman: Yes, absolutely. Think of it as being pulled over in your car and the cop is telling you you were doing 100 miles an hour even though your speedometer was saying you were doing 60 miles an hour. And so somebody might have tampered with your speedometer, and you're thinking you're going under the speed limit while the police officer with his radar gun picked you at 100 miles an hour. So, yes, what we're showing on the HMIs can no longer be just accepted or no longer trust it to be valid. And that's a big, big problem because now the operator sees the wrong values, and you might make the wrong decisions based on what he's seeing. 

Dave Bittner: And you went down the path in your research here to actually sort of proof of concept this man-in-the-middle technique. 

Pascal Ackerman: Yeah. So after I saw those clear text values passing by us, which was a nice finding on itself already, I thought, what if I could manipulate these values as well and start showing something else on the screen? And that's exactly what I did. I used, again, the Scapy tool to find a package that were interesting, look at specifics on how to change some of the values on there. The framework even makes it easy for me to - once I'm done with manipulating the packet, it will do a revalidation. It will do a fix up of the CRC in the packet, and then we'll send it off to the client. 

Pascal Ackerman: And hold and behold, I saw the values change on the HMI. So I was looking at the PLC itself, which is the source of the data. It was showing 100 degrees. And then the server - in the server-client HMI setup, it was showing 100 degrees. But by the time I was done manipulating the packet, the HMI was showing 2,000 degrees. So there was just for compromise of integrity. 

Dave Bittner: And you're able to do this in a way that wouldn't automatically draw attention to itself, would delay detection? 

Pascal Ackerman: It would be - so these attacks are all based around the ability to sit - to put an attacking machine, what my colleague Linux (ph) in the write-up, between the server and the client. In order to do that, we have to perform what's called ARP spoofing, which is pretty - it's really easy to detect if you have the right tools in place. So if you have, like, a SIM or a deep packet inspection tools installed, it will be detectable. But a lot of these facilities don't have those means in place yet. 

Dave Bittner: You know, as I was reading through your research and I came to the section that talks about the potential implications, I have to admit perhaps my twisted sense of humor was triggered. The first thing you list is panic (laughter). And while it's easy to laugh at that, I mean, the reality is is that in industrial environment, that's a real problem - people responding to something that is out of bounds for what they're used to. 

Pascal Ackerman: Yeah, yeah. And it's valid too because I have worked in controls and automation since 1999, and I've been out on the production field. And a lot of times these processes that have high-pressure steam, high temperatures are quick and oftentime (ph) hard to control. An operator has to be on his toes to keep it up and to keep it going. So if anything like this happens that is way out of whack, then it might have him push the emergency button which stops the process and would - all the consequences from doing that. 

Dave Bittner: Were you surprised to see so much data being exchanged in the clear? 

Pascal Ackerman: No, I - no. As I have been advocating for years now, industrial control system security was an afterthought, and that wasn't anybody's fault. It's just how it organically grew, right? When we first started seeing industrial control systems, there weren't more than a handful of devices that were tied together with serial communications, point-to-point connections. So authentication authorization wasn't in play at those time because there was only one device that was able to connect to it. But then when the hype came to bring everything to TCP/IP stack and put everything on Ethernet in order to be competitive and to easily have customers convert from point-to-point to serial to Ethernet, they just took those wide open protocols, and they put them on Ethernet, which was already being scrutinized by attackers with Wireshark and I think it was Ethereal at that time, tools, and the other common IP attack tools. 

Dave Bittner: So what is to be done here? I mean, what are your recommendations, given everything that you've discovered here? 

Pascal Ackerman: Ideally, I'd love to see the ICS manufacturers, the controls and automation vendors, start to build in authentication or at least integrity between the parts of their offerings where this kind of communication takes place. So in the case of Rockwell Automation, it would be great if they could set up a secure channel between their server and their client. Until then, we're going to have to rely on compensating controls, we call that, because you can set up a tunnel between the client and server operating systems. So Windows will allow you to set up something called an IPsec tunnel, which encrypts all communications between the client and the server. 

Dave Bittner: Yeah, it strikes me, too - and correct me if my understanding is wrong - that more and more organizations are relying on these kinds of remote sensors, you know, where you used to have a - perhaps a person on site at a location or, you know, on the other side of a plant or the other side of the - of a nation. And more and more of this is happening remotely, both for convenience and cost savings. 

Pascal Ackerman: Yeah. And oftentimes, they use what I call a SCADA system for them. And they will have remote channels. And they will have plans all across the world, and they all tie that data back to a central place. The good news is, though, that they often do that communication, that centralized model, they do that over what they call peer-to-peer VPN connections or some hub spoke VPN architecture. So from the internet, from the public network, this is distributed (ph) off by the VPN connection. But once you're on that network, all of this data is visible. 

Dave Bittner: Now, what you have outlined here is a proof of concept. Are you aware of anything like this happening in the wild - of any bad guys trying to take advantage of this sort of thing? 

Pascal Ackerman: No, not yet. 

Dave Bittner: Yeah. Seems like it's a matter of time (laughter), sad to say. 

Pascal Ackerman: It is sad to say. But luckily or unluckily, I don't know, attackers are still focused on ransomware. So I've read some research from a colleague researcher over - they had set up a - an ICS Honeynet where they basically open up some ports, and they made it look like they had an industrial control system sitting on the internet. And he stated that over the six or seven months they had that open, they had no really targeted attacks on the ICS systems, but mostly it was ransomware or malware trying to take over the network. So until that changes, stuff like this is - it's probably not that critical to look after yet. 

Dave Bittner: Our thanks to Pascal Ackerman from GuidePoint Security for joining us. The research is titled "GuidePoint Security Researcher Discovers Vulnerability in the Integrity of Common HMI Client-Server Protocol". We'll have a link in the show notes. 

Dave Bittner: The CyberWire "Research Saturday" podcast is a production of N2K Networks, proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. This episode was produced by Liz Irvin and senior producer Jennifer Eiben. Our mixer is Elliott Peltzman. Our executive editor is Peter Kilpe. And I'm Dave Bittner. Thanks for listening.