Joe Weiss

Managing Partner, Applied Control Solutions, and Managing Director, ISA99

Cyber Security for the Power Grid — Why We Should Fear Hackers (but not Squirrels)

January 14, 2016 — Joe Weiss, industrial control systems security expert and Managing Partner of Applied Control Solutions, talks about the challenges of securing the electrical grid, and why concerns about cyber attacks on industrial control systems are more than fear, uncertainty, and dread.

Be sure to catch the Week-in-Review podcast featuring Joe Weiss.

The CyberWire spoke with Applied Control Solutions' Joe Weiss, an industrial controls systems security expert who's also the Managing Director of the ISA99 standards body. We asked him about the December 2015 cyber attack on the power grid in Western Ukraine, and the lessons both cyber security and controls system specialists should draw from it. He put the incident in context and explained to us why indeed people should take cyber threats to control systems very seriously. Here's what he had to say.

The CyberWire: Thanks for talking to us today. We're interested in hearing your thoughts on the recent incident in Western Ukraine. Rolling blackouts, widely assumed to have been the work of hackers (probably Russian hackers, whom iSight Partners has identified with the Sandworm threat group, although attribution is of course always a dicey business) struck the Prykarpattyaoblenergo utility around Ivano-Frankivsk. What can you tell us about this episode?

Weiss: Thank you for giving me an opportunity to provide my thoughts. I think it's important, because there've been many discussions about what's occurred, and some of them don't really appear relevant to the situation. I'd first like to stress that analysis of the Ukrainian cyber event is still on-going. There's not a complete understanding yet of precisely what has happened. And, without trying to put too much importance on this, there's a parallel between what's happened here, and the reporting on it, to what happened years ago when Stuxnet was first found.

Stuxnet was discovered, if you will, because of the Microsoft zero-days. And it took quite a while for people to really understand what Stuxnet was actually attacking. IT was, as usual, focused on the IT aspects of Stuxnet, and I think you have a similar situation here. The real issue is that substation breakers were opened in a series of substations, and this in turn led to a three-to-six hour power outage that affected some 80,000 customers. That's what we should be focusing on. The hacking questions need to be related to the question of how the breakers were opened in the substations.

The CyberWire: We've heard well-attested reports that the incident is associated with distribution of the BlackEnergy malware kit. We've also heard that the attack was highly coordinated, and that the attackers, whoever they were, used a form of telephony denial-of-service to perform some misdirection and divert people from the attack. What do you think we know, at this point, and where do you think we should be looking as the investigation goes forward?

Weiss: From everything I've read, this appears to have been a coordinated attack, because it wasn't just malware. As you mentioned, there was a telephony aspect to the attack as well. BlackEnergy was found, but I'd refer you to a notice from DHS, ICS-CERT, about BlackEnergy. Black Energy's been around since 2011. Until now, BlackEnergy has been a data exfiltration tool. It hasn't been an attack mechanism. So I return to this point: we're still in the process of trying to understand what truly led to the breakers' being opened, which is what caused the actual electrical outage. We know there was malware in the system. The question is what tie, if any, malware had to the outage.

The CyberWire: Some of the reports we're seeing talk about the breakers being cycled not by malware proper, but rather because someone obtained persistence in the network and gained access to the controls. Does that sound plausible, based on what you know?

Weiss: From what I've read, somebody got access to not just one or two breakers, but to multiple substations, because an entire series of substations and breakers were affected. And it doesn't appear that that was done manually and locally, but that it was somehow done remotely. So this is one of the open issues: how was it done remotely?

I'd like to make one other point, too. When you get access to the breakers in the substation, there are a many things you could do to cause damage. However, there does not appear to have been equipment damage in this case. Essentially all the attackers did was open the breakers. It's like opening the fusebox in your house: once you reclose the fuse, the lights go back on. And the lights in Ukraine went back on in three to six hours.

The CyberWire: There was an interesting post by the industry group Foundation for Resilient Societies, in which they point out the significance of the attackers' hitting distribution substations. And you've said distribution substations are overlooked in electrical utility regulations. Does that seem correct to you?

Weiss: Let me make that statement clearer. The attack affected what's called "low-voltage transmission and electric distribution." Low-voltage transmission and electric distribution are excluded from the NERC1 Critical Infrastructure Protection (CIP) standards, which means that the targeted substations, if they'd been in the US, would not have been required to have any cyber security protection or evaluation because they would have been outside the scope of the CIP standards. It does not make sense that distribution substations that can affect 80,000 customers do not have cyber security requirements, and that's what the Foundation for Resilient Societies is pointing out.

The CyberWire: Let me ask you about the big picture here. Everybody is always concerned about FUD—fear, uncertainty, and dread—or hype, if you will, in stories about cyber security. A number of people have pointed out—and I think they're quite correct—that squirrels have been responsible for several orders of magnitude more power outages than have hackers.2 That's certainly fair enough, but could what we're seeing in Ukraine amount to something like the patient zero of a pandemic?

Weiss: Let me phrase it differently. And I think this is an important point, because this issue about squirrels has been brought up a lot. Consider this—if the worst impact you could manage with a cyber attack against the electric grid were a two- or even a three-day outage, that would really, in the grand scheme of things, not be that significant. Because not only do you have squirrels, but you have ice storms, hurricanes, earthquakes, and other events that disrupt power grids for short periods of time. So yes, of course squirrels can be a problem, but the reason we care so much more about the cyber security of the grid—and the reason I point out that in Ukraine the attackers chose not to cause real damage—is that with cyber you can bring the grid down for a significant period of time—many weeks to months or even years. That's something a squirrel can’t do.

The CyberWire: Nine to eighteen months would take a pretty big squirrel. There was an article in Wired on research that describes a novel, and reportedly relatively easy, way of remotely attacking variable frequency drives. Is that the sort of damage you have in mind?

Weiss: Remotely hacking variable frequency drives can have significant impacts on large rotating equipment in power plants, refineries, dams, etc. The impact can be short-lived, like the manual shutdown of the Browns Ferry 3 nuclear power plant, or more devastating, like the destruction of the Sayano-Shushenskaya dam in the Soviet Union that killed 75.

The CyberWire: You've also been involved with the Aurora Project, an experimental study of the vulnerability of power generation equipment to destructive hacking. Does Aurora have any lessons you're able to share?

Weiss: Aurora is a physical gap in protection of the electric grid. Aurora is caused by opening a breaker and then reclosing it out-of-phase with the electric grid. Aurora requires specific hardware mitigation to prevent major equipment damage. However, very few utilities in the US have installed the requisite hardware mitigation. The Ukrainian hack opened breakers, which is step one of the two steps in an Aurora attack. Yet the E-ISAC in a press release said the Ukrainian attack could not occur in the US, and there was no need to change any policies even though the substations attacked would have been out-of-scope of the NERC CIPs. It should be obvious something is amiss.

Less than a month ago, DARPA—that's our Department of Defense—issued a Broad Agency Announcement (BAA) for a major new project on the cyber security of the electric grid. That’s an acknowledgement that cyber security of the electric grid is still a very big problem. There have been many control system cyber incidents in electric grids, pipelines, transportation, chemical plants, and even in nuclear plants. However, because of a lack of adequate control system forensics, most of those incidents weren’t recognized as being cyber-related.

The CyberWire: Let me close with two questions, giving you an opportunity to offer advice to people in two different sectors. Most of the people who work in cyber security come from an information security background. What advice would you have for them about the security of industrial control systems? And the second, related, question is what advice would you offer electrical utilities?

Weiss: First, industrial control systems are composed of two elements. The first is the human-machine interface, the HMI, which is generally Windows-based, and that's where the IT security community can play a big, immediate role. But the second part, which is where we saw the problem in Ukraine, involves the actual control system devices that monitor and control systems in real time. And these generally are not Windows-based. They're also often not on Internet protocol networks, and they are very, very different from the HMI. The whole point of why the International Society of Automation (ISA) formed ISA99, to develop industrial control system cyber security standards, is because securing control systems in all industries is very different from securing business IT systems.

There has to be a joint effort between the security people who understand IT—but do not understand the domains of electric power, water, chemicals—and the engineers who understand that domain, but may not understand security. And one of the biggest problems today with securing industrial control systems is that the IT community is running amok without understanding the impact of using inappropriate policies, technologies, and testing to secure industrial control systems. Consequently, there needs to be a joint effort between the people who understand the domain and the people who understand security, and that isn't happening yet.

As to what you'd tell the electric utilities, you'd tell them the same thing you'd tell the water companies, the chemical companies, the transportation services, etc. You'd tell them their systems were not designed to be cyber secure. They weren't even designed to be accessed remotely in a secure way. What the operators need to know—and here I mean not the IT people, but the control systems people—is that cyber threats can affect the reliability and safety of these systems. In general—and too many people think this is just FUD, but it's not—cyber attacks can damage equipment, kill people, and lead to major environmental releases, and these things have already happened. I'll leave you with this—there've been more than a thousand deaths to date from control system cyber incidents. That's why it is so important to address control system cyber security.

The CyberWire: That's disturbing. Can you tell us about one or two cyber events that have caused loss of life?

Weiss: Most of the control system cyber incidents that have injured or killed people have not been malicious. Unfortunately, that may be changing. The NIST definition of a control system cyber incident is electronic communication between systems that affects Confidentiality, Integrity, or Availability (C, I, or A). In the IT community C is most important, while in the ICS community, it's A and I. (Ironically, the most important aspect is what isn’t even addressed – Safety.) An example of a control system cyber incident that killed people was the 2010 San Bruno natural gas pipeline explosion that killed 8 and destroyed a neighborhood. The explosion occurred because of the overpressure in the pipe. The overpressure was created as a result of the maintenance performed on the SCADA system and the resultant signal to open a control valve. Even though this incident was not malicious, there was a programmable logic controller (PLC) at the SCADA center that, with changing some code, could have maliciously caused the same overpressure situation.

We have to address it in the right way, and I don't believe most of what's happened so far has been the right way.

The CyberWire: Thank you, Mr. Weiss.

1North American Electric Reliability Corporation.

2See, for example, this engaging piece in Naked Security, "CyberSquirrel 1: what you need to know."