Late yesterday the Sheriff of Pinellas County, Florida, said that his office was investigating an attempt on Friday to alter chemicals introduced into the city of Oldsmar’s water supply. An unknown party had remotely accessed the water utility’s control systems and directed that the amount of sodium hydroxide be increased by a factor of a hundred, from the safe, intended concentration of 100 parts per million to a dangerous 11,100 parts per million. A treatment plant operator noticed the change and immediately corrected it. The Tampa Bay Times says the authorities have some leads, but that no arrests have been made.
Water-treatment chemistry and sodium hydroxide.
Sodium hydroxide, familiarly known as lye, or caustic soda, is a strong base that's the principal ingredient in many paint-stripping and drain-opening products, and (less scarily) in many soaps. It's used in small quantities to regulate the acidity of drinking water (and in even smaller quantities it's used in cooking—curing olives, preparing lutefisk, baking German pretzels, u.s.w.). But it's a highly caustic and dangerous chemical in high concentrations, and so this is a serious attack that could have had lethal consequences. Pinellas County officials stressed that there was no danger, that it would have taken twenty-four to thirty-six hours before the sodium hydroxide concentration reached dangerous levels, but the incident is nonetheless a frightening one.
No attribution of the Oldsmar cyberattack, yet.
Despite a fair amount of tweeting and woofing about acts of war and so on, there's been no attribution of the attack. The operator who stopped the attack noticed something was amiss when his mouse cursor began moving. Jorge Orchilles tweeted a lesson from the world of penetration testing: "The easiest way to get caught as a red teamer is to move someone’s mouse. Nothing freaks people out more than their mouse moving when they aren’t touching it. It is a psychological thing." Kevin Collier thinks this suggests that the attacker is probably more skid than mastermind, tweeting "We know almost nothing about who they are, but here's a strong indication this wasn't a masterminded plan." That's not necessarily reassuring, he added, "Is it comforting to know this probably wasn't some Russian master plan to poison some Floridians? Or more disturbing to think this is how close an amateur could get?"
It is, however, important to emphasize that nothing is publicly known so far about who may have attempted the attack. It's also worth remembering that the simplicity of an attack, its ease of execution, says little more than that there's a broad range of threat actors who could have accomplished it. In this case that ranges from a failed-to-launch skid in the parents' basement all the way to a nation-state's espionage or military services, from a chump down the block doing something for the lulz up to one of Huggy Bear's brood.
Vulnerabilities and incident response in the water treatment hack.
The attacker is believed to have obtained access to the water treatment plant's TeamViewer software, WIRED reports, adding that the city disenabled TeamViewer shortly after it noticed the attack. TeamViewer is also relatively easy to use (and it can be accessed with stolen credentials) and some have seen this as another indication that the attack was not a sophisticated one. Bryson Bort, founder and CEO of SCYTHE, commented in an email that, “TeamViewer is a common remote desktop protocol (RDP) solution in ICS and the water attack was most likely simple access with stolen credentials. Using the software means everything is visible to the user (hence, the operator saw the mouse move and settings changed). Who and why is still the question.”
Niamh Muldoon, global data protection officer at OneLogin, in emailed comments gave Oldsmar's water utility relatively good marks for its response: “After the event has happened, crisis management is critical for successfully managing the attack response to reduce business impact and consequences, and it appears the Florida agency has done that."
Recent historical precedents: wastewater in Israel, a small dam in downstate New York.
What have other control system attacks looked like? Last spring Israeli authorities warned that Iranian operators made an attempt on water treatment and wastewater facilities in two rural districts in Israel. They weren't fully successful: the Council on Foreign Relations has a summary of that incident on its site.
There was another incident in which the controls of a small flood-control dam in Rye, New York, were remotely accessed. In 2013 the Bowman Street Dam's controls were accessed. The US would ultimately indict an Iranian cyber operator for that action.
Austin Berglas, former head of FBI NY Cyber and currently Global Head of Professional Services at BlueVoyant, led the investigation into the Rye incident, and he offered us some perspective:
"Along with energy production and manufacturing, water supply facilities are part of the United State’s critical infrastructure and have long been targets for cyber attack from both criminal and state sponsored entities. Water facilities rely on systems control and data acquisition (SCADA) systems to manage the automated process or water distribution and treatment. Many of these industrial control systems (ICS) are outdated, unpatched, and available for review on the Internet, leaving them incredibly vulnerable to compromise. In addition, many ICS solutions were designed for non-internet facing environments and therefore did not incorporate certain basic security controls - this offers additional vulnerabilities as more and more operational technology environments are allowing access to their ICS systems from the Internet. In 2013, the FBI investigated a compromise of the Bowman Avenue Dam in Rye Brook NY and found that members of the Iranian Revolutionary Guard had gained access through Internet facing controls. Although the Dam was not functioning at the time and was most likely not the Iranian’s main target, it demonstrates the vulnerability of certain critical infrastructure when their ICS systems are allowed to be exposed to the Internet and not isolated. (See the New York Times on the Bowman Street Dam hack.)"
Many have commented that leaving the supervisory controls of a water treatment system open to remote access is extraordinarily risky. (See, for example, comments to that effect by TechCrunch's Zach Whittaker.) Such systems had long been relatively immune to cyberattack because their age and the legacy control systems they employed effectively air-gapped them. Berglas, however, sees such connectivity as a foreseeable aspect of modernizing systems. "Digitization and IoT expansion have allowed for previously isolated infrastructure to be accessed remotely," he wrote. "For example, water and utilities need to balance security while allowing operators the ability to remotely access treatment plant SCADA systems from phones, work, and computers—in order to react to alarms and respond to incidents without having to be physically onsite."
As was the case with the Florida incident, no real harm was done by the Bowman Street Dam hack. Berglas thinks it likely that the attack on the small sluice gate in Rye just afforded a "proving ground to test capability and techniques." And, again, when asked about possible attribution of the Oldsmar attack, he sensibly said, simply, "Too early to tell."
A cyberattack as an act of war?
This wasn't obviously a financially motivated crime, nor was it, assuming a nation-state was behind it, mere espionage. It's best to regard it as an unsuccessful (and in some ways pretty casual, since the attackers were only in the system, it's believed, for about five minutes) attempt at sabotage. Sabotage, especially unsuccessful sabotage, seldom amounts to a casus belli, and Dragos CEO Rob Lee tweeted much the same conclusion.
Responding to a control system cyberattack.
Dragos's Lee also cautioned against both premature speculation about attribution and thinking that a challenge like this could be addressed with any single, simple solution. It's a systemic problem with many interdependent aspects. "Hiring, workforce development, culture shifts, working within national priorities and regulations, state and local regulations, resourcing other areas that are organizational challenges, modernizing infrastructure beyond "cyber", etc. There's not 1 easy answer tech or not."
It's troubling, for example, to think that in this case the safety of a water supply depended upon one watchstander happening to notice that something was briefly unusual on his screen. Dragos has published a set of considerations and recommendations other utilities might well consider.
Reaction in Congress.
US Representative Jim Langevin (Democrat, Rhode Island 2nd District), a member of the House Committee on Homeland Security's Subcommittee on Cybersecurity and Infrastructure Protection, tweeted that the incident is another reminder that the Internet wasn't created with security in mind. US Senator Marco Rubio (Republican, Florida) tweeted that the incident should be treated as a "matter of national security," and that he's requested an FBI investigation, which is surely already in progress.