Research Saturday 9.4.21
Ep 199 | 9.4.21

Like a computer network but for physical objects.


Dave Bittner: Hello everyone, and welcome to the CyberWire's Research Saturday. I'm Dave Bittner, and this is our weekly conversation with researchers and analysts tracking down threats and vulnerabilities, solving some of the hard problems of protecting ourselves in a rapidly evolving cyberspace. Thanks for joining us.

Ben Seri: In many ways, it's very similar to a computer network, just for physical objects.

Dave Bittner: That's Ben Seri. He's VP of Research at Armis. Today we're discussing their research on remote code execution vulnerabilities in Swisslog's pneumatic tube system.

Ben Seri: You have stations which are the endpoint of the system, where staff at the hospital can send and receive carriers. And these stations are connected with the tubes. And there are routers, and they're called "transfer units" or "diverters" in the terminology of pneumatic tubes. And these literally shift carriers within the various intersections of tubes. And then you have blowers. These are what maintains the air pressure within the system, and they can pull or push through the air depending on the direction of the carrier. So this is really, in many ways, a network of physical transports, physical objects. similarly in ways to how packets are passing through an IP network.

Dave Bittner: You know, I can't help but be reminded of, you know, years ago, there was – people were making jokes about comparing the Internet to a series of tubes. And here we have an actual series of tubes. (Laughs)

Ben Seri: Right, right. Yeah, it is. It is a series of pipes. There is also, in computers, pipes are a way to – within a Linux system, for example, pipes are a way to transfer data between processes. And here, there are literally pipes are used to transfer physical objects.

Dave Bittner: Well, help us understand what's going on to control all of this. I mean, obviously, there's routing. You've got to get things from point A to point B. For the folks who are using this system, what's going on under the hood?

Ben Seri: Yeah. So, maybe unlike the IP network, in this case, this has to be orchestrated very carefully by a central management server. When you want to do a transaction of a carrier through the tubes, you first have to align all of these diverters to the correct path of the tubes. So you would create a link between a source and destination station. Then you would need to turn on the blower and have the air pressure set to the correct speed. There is a feature within these systems that you can control how fast you send a carrier within the tubes. And this is important to have, specifically because certain items like blood products, they can be damaged if they're sent too fast within the tubes. So you can control – you have slow transfers and then you can have urgent transfers for stuff that you need to deliver really quickly. So, when you set up the tubes with the diverters, when you turn on the blowers, eventually the station's door opens, and then the staff can put in the carrier and then it would be whisked off within the tubes.

Ben Seri: It's a system in which you have one transaction at a time per what is called a "zone." And a zone is a part of the hospital where the tubes interconnect in a way that can only transfer one carrier at a time. But then you can have multiple zones and they will interconnect with inter-zone diverters. So that's a way to make more transactions occur at a time. But essentially, this is a complex network of analog components that needs to be synchronized to allow transactions to take place.

Dave Bittner: And to what degree is this automated? Are there humans keeping an eye on things, or does it pretty much run itself?

Ben Seri: It pretty much running on itself. I would imagine that back in the day, maybe when you were a child in the department store, it would have much more analog and manual management, like elevators had in the back in the day, right? So there would be an operator. And so, I saw an image, for example, that you see this type of system was installed in offices where it was used as inter-office messaging. And then in large offices where you had, I don't know, hundreds of these stations or endpoints where you can load carriers, you would have a physical operator, and you would put in the carrier, it would be sent to the operator with a destination written on it or something of that nature, then the operator would send it to its destination. So, very much like how telecommunications, how phones worked earlier on.

Dave Bittner: Hmm.

Ben Seri: But yeah, that's back in the day. Today, everything is managed automatically. It does everything – the central management server does all the coordination automatically. There are some maintenance to be done. Sometimes issues can occur. Swisslog specifically – the company, the vendor that we found vulnerabilities in its product –does offer a service for hospitals in which it manages and monitors their central management server remotely from the Internet, which is probably a good feature to have, so you don't need someone physically monitoring it at each hospital. But it also creates an attack surface, right? Because the central management server now needs to be connected to the Internet. The Internet is an obvious attack surface, and also that connectivity to the monitoring solution by Swisslog, if it's found vulnerable in the future, then that can be an entry for attackers to take over the PTS network from the Internet.

Dave Bittner: Well, and this leads us to the research that you and your colleagues have published here, where you all have discovered some vulnerabilities in this systems. How did this first come to your attention?

Ben Seri: So, from time to time, we do proactive research. My team has all kinds of tasks within Armis, but part of it is really looking at our customers' environments, understanding what are the most common and most critical types of devices that they use, and doing research on it and trying to find vulnerabilities. And so in our healthcare customers, I noticed the fact that all of them use this pneumatic tube system, which I was not aware before that existed. And for me, it was, you know, a blow from the past to see something like that actually in use, and it was just very popular.

Ben Seri: So, the Swisslog system is used in over eighty percent of hospitals in North America. It's installed in over three-thousand hospitals worldwide. Every major hospital needs to have pneumatic tube system, and Swisslog is the leading vendor in North America for this type of solution. So it was apparent to us that this was very popular, on one hand. But on the other hand, it's very not known – it's little-known to the general public. And they probably haven't received any research efforts because of that, because it's just hidden within the walls in hospitals and people don't think about it.

Dave Bittner: Right. Well, take us through what you discovered here. What exactly is the vulnerability?

Ben Seri: So, it's nine vulnerabilities. And they are critical in nature, because they can allow takeover of the station within this network. The current models of this Swisslog Translogic PTS stations are all based on a board called the "Nexus Control Panel." And this control panel runs Linux, and it is the brains of all the current station models by Swisslog. And so, we're having an access to the hospital network where these are installed, and these are IP-connected. An attacker can take over them with various different vulnerabilities that we found.

Ben Seri: So, there was a Telnet server open on this device, unfortunately, that wasn't supposed to be left open in production. It has hardcoded password that we are able to find. These can be used to log in to the device and take over it. There is also a privilege-escalation vulnerability that can allow root access. So that's one bunch of vulnerabilities. Then there are a couple of memory-corruption vulnerabilities. And these are in the protocol that manages these stations, the protocol that the central server has with all of these stations. And they can also reach a remote code execution. There is also a denial-of-service vulnerability we found. And lastly, a very serious design flaw in which the firmware upgrade process of these devices is completely not secure. So, the firmware is not signed, it's not encrypted, and there isn't any authentication needed to trigger the firmware update process.

Ben Seri: So, all of these different vulnerabilities can allow attackers to compromise the Nexus Control Panel, which powers all current models of the stations. And that's – it only requires access to the network, which is something that the attacker would need to have to trigger the attack. But it does not require authentication. It doesn't require end-user interaction. It's a remote attack in various ways.

Dave Bittner: What has the response been of Swisslog? Have you reached out to them?

Ben Seri: We have, yes. So, we've been in contact with them since the beginning of May. It took some time for them to understand that this was a serious issue and that they need to handle it. This is the first disclosure that they're experiencing. They're a very serious company, and they have a very advanced deck, but they come from an era which is an analog era in many ways, and so the security was maybe not completely part of the design of these systems. But they have gone through the path of understanding the vulnerabilities better developing a patch. They have released a patch on Monday – so on August 2nd, when this disclosure became public, and a security advisory in which they offer various mitigations. And Armis has also published a security advisory detailing how this can be blocked by various tools, and how to mitigate the risk in the best way possible.

Ben Seri: So, this is a very important looking forward. What we found maybe is now receiving a patch that we would fix the specific issues. But we believe that the system is now being researched in a more wider way, other issues would be found in them. And it's important to harden the access to them as much as possible.

Dave Bittner: And what are the potential issues here? What are the dangers of a system like this being taken advantage of?

Ben Seri: Yeah, so, first, I think it's important to understand how critical it is within the hospital. So, what is its actual tasks? And it starts from the fact that testing within a hospital, lab tests, there is this daily motion that a hospital needs to do on a regular basis all the time. So, to automate this process, all of the departments, all of the nursing stations have these stations, and lab samples, various specimens now that are taken from patients are sent through the tubes to a central laboratory where it is tested quickly, and then the patient care is continued based on that test. So that's one use case, which is very, very common.

Ben Seri: There are other pieces of the PTS network within hospitals. Pharmacies within hospitals usually connects to the to the PTS network, where they distribute medicine to all of the departments using this network. The blood bank in the hospital might be connected to this network so blood units can be sent to operation rooms from the blood bank. And so, there are various applications in which this is used. And again, critical items such as blood units and various specimens are shipped within the network.

Ben Seri: So, just understanding the fact that this is a critical infrastructure, and if it were to be shut down unexpectedly, this would result in some effect on patient care. In a way, just because hospitals are this chaotic scene by nature, and adding more chaos to that scene is something that can harm patient care services. This is why any attack on this system can have consequences just by the fact that this system is so delicate and so critical to the operations of a hospital.

Ben Seri: But then there are also other elements of why an attack on this network could be meaningful for an attacker. So, this – the PTS solution by Swisslog, it integrates with other hospital infrastructures, and that can hold some sensitive data within these integrations. So, for example, the access control system of the hospital that manages physical access to doors by authenticating an RFID card that the nurses and the doctors have – this system is usually integrated with the Translogic PTS solution to allow the staff to be authenticated with the PTS stations, so only the staff can use the pneumatic system and not some patient or anybody else as they're walking through the corridors. And that type of integration exposes the RFID credentials of the staff, staff records, stuff like that to any attacker that takes over the system, because this sensitive data passes through the system.

Ben Seri: So, all kinds of attacks are possible on the system. One would be an information leak. Another would be shutting them down, which would be harmful for hospital. And maybe the third most sophisticated type of attack – it is not likely to occur by a simple attacker, it would require a more sophisticated attacker – but it is possible to abuse this system in a way that derails hospitals operations until they understand that the network has been compromised.

Ben Seri: So, just doing a man-in-the middle attack on the system in which you change the paths of the carriers. An attacker sitting on the station, compromising the stations, can intervene with the correct path that the carrier should go through, and that would create more chaos in the hospital. He can change the speeds in which the carriers travel through the tubes. And as I mentioned, some items are sensitive to the speed of which they travel. So that can damage their content. And all of that might be used by a very sophisticated attacker to seek a ransom. So, just holding the network of the tubes hostage until a payout is made.

Ben Seri: So, it's not something that we usually connect to a ransomware attack – we know that ransomware attacks are normally connected to PC endpoints and their files being encrypted. But essentially, ransomware is just the use of something sensitive being taken hostage. And the network itself, the pneumatic tube network, can also be taken hostage. As I mentioned, one of the vulnerabilities we found is the fact that the firmware upgrade process of this device is very much not secure, so an attacker is able to – can maintain persistence on these devices. So once he's done that, it will be very hard to get rid of him if he demands a certain payout for them for them to stop using this attack.

Dave Bittner: So, is the message here, I mean, I suppose there are plenty of people in our audience who have customers, colleagues, and so forth in the cybersecurity realm who are either working with hospitals or hospital-adjacent or suppliers of hospitals – I suppose a big part of this is just spreading the word that this vulnerability exists and that there are mitigations in place that people should take a serious look at.

Ben Seri: Yeah, I agree. And I think that – so, we also have healthcare customers, and when we brought this news to them, it was also apparent that they, too, were not completely aware to the fact that this system is in use at their hospitals and that it is so critical. So, it's just something that is hidden within the walls, it works, you don't think about it, you don't pay any attention to it. And so security aspects of it are also not in front of you, or not something that you're thinking about.

Ben Seri: So, it's a two-fold process. First, raising the awareness of the fact that these systems exist. They're important. They're critical. And second, that they are vulnerable, and there are ways to mitigate risks around them and to better protect them.

Ben Seri: I think just in a more broader term, it's important to understand when you think about healthcare security, that it starts from the medical devices – the life support systems, infusion pumps, the stuff that are directly connected and are providing their function to the patients – these devices, security is very important, obviously. But then you should look at the hospital in a more holistic way and understand that there are other systems involved in providing the patient care. Maybe they're not categorized as medical devices. Maybe they're these transport systems with pneumatic tube systems. Maybe they are the electricity of the hospital, water or irrigation systems, the elevator. But there are all of these systems that interconnect and are eventually what allows the hospital to provide its patient care and provide the best service that it can. So, the security community, and for the healthcare space, looking at the attack surface in a more broader way, I think this is a very good way of moving forward.

Dave Bittner: Our thanks to Ben Seri from Armis. The research covers remote code execution vulnerabilities in the pneumatic tube system of Swisslog. We'll have a link in the show notes.

Dave Bittner: The CyberWire Research Saturday is proudly produced in Maryland out of the startup studios of DataTribe, where they're co-building the next generation of cybersecurity teams and technologies. Our amazing CyberWire team is Tre Hester, Elliott Peltzman, Puru Prakash, Justin Sabie, Tim Nodar, Joe Carrigan, Carole Theriault, Ben Yelin, Nick Veliky, Gina Johnson, Bennett Moe, Chris Russell, John Petrik, Jennifer Eiben, Rick Howard, Peter Kilpe, and I'm Dave Bittner. Thanks for listening. We'll see you back here next week.