Water Sector Prepares For Cyberattacks

Security threats evolve as water systems connect to the internet.

Industrial control systems, which operate the pumps, valves, and disinfection systems at drinking water treatment plants, are at increasing risk of cyberattack. Photo © J. Carl Ganter / Circle of Blue

By Brett Walton, Circle of Blue

Rye, New York, a wealthy commuter town a 50-minute train ride from central Manhattan, does not seem a typical target for a cyberattack. It is not a major retailer like Home Depot or Target, both of which had millions of credit card accounts stolen in recent years. It is not a military base. It is not home to an important electric plant or substation. Only 16,000 people live there.

Yet for three weeks after Labor Day in 2013, the city’s computers were exposed to prying eyes. An Iranian computer expert with links to the country’s spy agencies repeatedly hacked into the system that operates Bowman Dam, a small structure hardly bigger than a barn door that controls water levels on Blind Brook, a calm stream. Marcus Serrano, the city manager, told Circle of Blue that city officials did not know they were being hacked until they received a call from the Department of Homeland Security. The details of the attack are laid out in a Justice Department indictment that was unsealed in federal court on March 24, 2016.

Cybersecurity authorities refer to the Bowman Dam incident as “a read,” meaning that the hacker was looking for information about the dam’s operating system and not attempting to take over the controls. Rye officials do not know why their dam was targeted. They think it may be a case of mistaken identity.

The hacking of Bowman Dam, though not a typical cyberattack, is the most striking example of a problem that U.S. water utilities now acknowledge with greater clarity and vigilance. Computer networks and the industrial control systems that raise dam gates, operate pumps, disinfect drinking water, and guide an assortment of other critical functions are vulnerable to the digital version of breaking and entering. (see sidebar)

In the evolving realm of cybersecurity, where the risks today are different than just a few years ago and the threats tomorrow will be different still, a water utility’s size may not be the most important characteristic for those looking to do harm. “Sometimes cyberattacks are targets of opportunity,” Michael Arceneaux, managing director of WaterISAC, a cybersecurity information hub for the water sector, told Circle of Blue.

The opportunities for malicious activity are increasing. More and more water utilities, in order to save money by remote monitoring, are connecting their control systems to the internet. Meanwhile, hackers are developing computer viruses capable not only of stealing data, but also of taking control of critical infrastructure. “Cybersecurity is a huge and emerging public risk,” Daniel Groves, cybersecurity program manager at Arcadis, a consultancy, told Circle of Blue. “It’s growing more complicated and difficult daily. The attacks are becoming more sophisticated.”

The threats are real, according to cybersecurity experts working for government agencies and in the private sector. Ransomware, the most common cyberattack, according to Arceneaux, locks down computer networks or control systems and demands payment, usually in bitcoins. Though U.S. water utilities have been infected this way, none has paid a ransom, according to Bob Timpany of the Department of Homeland Security. They have recovered their operations by replacing the system or restoring a backed up version.

Other types of malware are more worrisome threats. Strains such as Havex and BlackEnergy, which was used in an electrical grid attack in December 2015 in Ukraine that shut power to an estimated 700,000 people for a few hours, are designed to take control of critical systems. In a worst case scenario, a water pump station could be overworked to the point that it blows the pipes out of the ground.

The National Infrastructure Advisory Council, a group of experts that advises the Department of Homeland Security and the president on critical infrastructure, says that cybersecurity awareness among water utilities is “often limited” and that the number of cybersecurity experts in the sector is “insufficient for current needs.” Those conclusions are included in a report on water sector resilience to natural disaster and cyberattack that will be released later this month.

For the U.S. water sector, the concern is directed more at the attacks that might occur, rather than the attacks that have already happened.

“To our knowledge, we have not seen anyone manipulate or destroy water infrastructure in the United States” because of a cyberattack, Timpany told Circle of Blue.

It Takes a Village to Rebuff a Cyberattack

Timpany’s official title is the operations chief of the Idaho Falls branch of the Industrial Control Systems Cyber Emergency Response Team, known in security circles as ICS-CERT. It is a division of the Department of Homeland Security.

ICS-CERT is a 9-1-1 switchboard, EMT unit, triage division, and family doctor in one government package. The division, which has a fly-away team for immediate on-site support, specializes in diagnosing computer viruses and rehabilitating infected industrial control systems, or ICS. ICS-CERT calls these systems “the pinnacle of cyber targets” because they have critical importance for the operator, and thus the highest value for the attacker. No mayor or governor desires a citywide blackout or a failure of the sewer system.

Utilities hope that a midnight call to ICS-CERT is never necessary. There is an expanding web of collaborators that are helping to prevent that from happening, or at least minimize the damage if it does. ICS-CERT, in addition to its incident response, conducts vulnerability assessments with utilities that request them. The number of assessments grew from 23 in 2013, to 38 in 2014, to 39 last year.

WaterISAC, another participant, serves as the cybersecurity information center for the water sector. It was formed in 2002, after the September 11 attacks focused attention on critical infrastructure. “We help utilities reduce risk by sending out guidance, checklists, and best practices,” said Arceneaux, the managing director. About 7,500 people who work at water utilities and state agencies are members, he reckoned.

Other groups are working on protocol. The National Institute of Standards and Technology, in response to President Obama’s Executive Order 13636 that was issued in February 2013, developed a cybersecurity framework to provide a common language to talk about cybersecurity risk. The American Water Works Association, the water industry’s main trade groups, is working to achieve widespread adoption of the standards among utility leadership by providing guidance on implementing them.

Do Not Open the Attachment

The basic problem for water utilities today is the convergence of two systems that used to be relatively segregated: information technology (IT) and operational technology (OT). IT is what a layperson commonly associates with cyber threats: the computer systems that are linked to the internet for email, billing, bookkeeping, and desk work. Viruses enter these systems through a mess of pathways: infected USB drives, email attachments, bad links on compromised websites.

The worlds of information technology and operational technology are merging, which adds new complexity for guarding water utilities against cyberattack. Photo © J. Carl Ganter / Circle of Blue

Basic cybersecurity starts with the individuals who work at the water utility, says Kevin Morley, security and preparedness program manager with the American Water Works Association.

“The human in the chair is the biggest vulnerability,” Morley told Circle of Blue. “They like to click on things.” Even an e-cigarette charged on a computer’s USB port could be an infection pathway, Morley said.

The OT side is traditionally the engineer’s domain. These are the industrial control systems so critical to drinking water treatment plants, power stations, and other big, centralized infrastructure. They are the targets of Havex, BlackEnergy, and a host of other nasties.

Utilities connect their OT systems to the internet to save time and money on monitoring costs. Remote access means a worker does not have to drive to each pump station to check its performance. Timpany does not tell utilities not to implement wireless connections. But he recommends doing so only if the encryption is secure, passwords are strong, and firewalls are in place between the IT and the OT.

Despite the efforts to prevent cyberattacks on water utilities, Morley argues that the sector needs to be realistic about its capabilities and limitations. Avoiding attack altogether is out of the question, he says.

“If the Pentagon is being hacked by the Chinese and the Iranians, I think we need to have a reality check about a water utility’s ability to stop a state-sponsored cyberattack,” Morley said. He then deadpanned: “A water utility does not have the resources of the Defense Department.” More important is avoiding “cascading consequences” among interconnected water, energy, and public health systems — the loss of a power station that cuts electricity to the water system, for example.

More modeling of potential scenarios could help. Timpany said that ICS-CERT demonstrated the destruction of an electricity generator through cyberattack as a learning exercise, but it has not done that for the water and wastewater sector.

More Awareness But Data Is Scarce

A rigorous accounting of the U.S. water sector’s cybersecurity preparedness is not possible, argues Timpany. Unlike, say, the United Kingdom, which has only 30 water providers, the United States has more than 52,000 public water systems, from big metropolitan networks that serve millions to community services districts that serve hundreds. Only 25 water utilities reported cyber incidents to ICS-CERT in 2015. An unknown number of incidents go unreported, Timpany said.

“It’s hard for anyone to truly articulate the risk in a big, grand statement for the United States water sector because there are so many systems,” Timpany said.

In general terms, however, the trend lines are moving in the right direction.

“The amount of awareness at the management level is much, much higher than it used to be,” Groves, the consultant, said. “I view that as encouraging, that management is starting to understand the risks.”

Morley, too, said that water utilities are showing interest in the topic that wasn’t apparent five years ago.

Yet change comes slowly. The water sector, on the whole, is a conservative industry, Groves said. That is understandable, given the essential service they provide. However, if water systems are going to avoid the digital traps of the 21st century, they need new awareness and new tools: technology to deflect attackers that are constantly probing for weakness; training for managers, particularly at small utilities, who will help integrate the OT and IT worlds; collaboration with vendors who sell the array of control system products; different skill sets for engineers used to dealing primarily with pipes and valves; and an appreciation that cybersecurity will never be solved, only managed.

“It’s an overall cultural change,” Groves explained. “We’re progressing toward it but we have a long way to go.”

Brett Walton

Brett writes about agriculture, energy, infrastructure, and the politics and economics of water in the United States. He also writes the Federal Water Tap, Circle of Blue’s weekly digest of U.S. government water news. He is the winner of two Society of Environmental Journalists reporting awards, one of the top honors in American environmental journalism: first place for explanatory reporting for a series on septic system pollution in the United States(2016) and third place for beat reporting in a small market (2014). He received the Sierra Club’s Distinguished Service Award in 2018. Brett lives in Seattle, where he hikes the mountains and bakes pies. Contact Brett Walton