Cleaning up SolarWinds Hack May Cost as Much as $100 Billion

This article originally appeared on Roll Call.

American businesses and government agencies could be spending upward of $100 billion over many months to contain and fix the damage from the Russian hack against the SolarWinds software used by so many Fortune 500 companies and U.S. government departments.

“Unlike good wine, this case continues to get worse with age,” said Frank Cilluffo, director of Auburn University’s McCrary Institute for Cyber and Critical Infrastructure Security. “For a lot of folks, the more they dig, the worse the picture looks.” 

Not only were at least four government departments targeted by the Kremlin hack — Commerce, Treasury, Homeland Security and Justice — but also thousands of top global corporations who were customers of SolarWinds, Cilluffo said. While government agencies appeared to be primary targets, “it doesn’t mean the private sector isn’t affected as well,” he said. 

The SolarWinds attack exposed 18,000 clients of the software management company after they downloaded and installed a tainted software update that was infected with malware. The breach occurred sometime between March and June this year and wasn’t discovered until cybersecurity research firm FireEye, which was attacked separately, revealed the SolarWinds breach in early December. 

After weeks of suggestions from former U.S. officials that the hack was the work of Russian intelligence services, the FBI, the Office of the Director of National Intelligence, and the Cybersecurity and Infrastructure Agency in a joint statement last week confirmed that it was indeed Moscow that was behind the attack.

The agencies said that the hack appeared to be “an intelligence gathering effort.” A much smaller number than the original 18,000 SolarWinds clients “has been compromised by follow-on activity on their systems,” the statement said.

Fewer than 10 U.S. agencies were potentially compromised by follow-on activity and the FBI and the intelligence agencies are “working to identify the non-government entities who also may be impacted,” the statement said. 

Espionage or something worse?

While the initial intent of the attackers may primarily have been espionage, they could change their motive, Cilluffo said. If the attackers are not fully eliminated from government and private company networks, they could choose to use their presence for more destructive purposes, Cilluffo said. 

Finding and eliminating the adversaries’ presence on networks is likely to be a costly affair, one expert said. 

“The reality is everybody is spending resources right now” on trying to figure out how far the hackers penetrated computer networks and how to get rid of them, said Jake Williams, a former National Security Agency hacker who’s now the founder of Rendition Infosec LLC, a cybersecurity firm. 

“The true cost could be hundreds of billions of dollars,” Williams said, when one considers the incident response cost for each breach multiplied by the 18,000 entities that fell victim. 

Government agencies and private companies also have to figure out if the network breach led to any loss of data and whether they have to alert Congress and customers as required by law, Williams said. 

Many private companies are discussing internally whether they should go public about being breached if there’s no evidence of any data being manipulated or stolen, Williams said. “There’s a lot of hand-wringing going on in the background, and companies don’t know what the next step is.” 

Austin, Texas-based SolarWinds developed and supplied network management software that top U.S. government agencies and Fortune 500 companies used to monitor their own networks. On its now deleted customer list page, SolarWinds claimed that its clients included 425 of the Fortune 500 companies including Microsoft, Lockheed Martin and Ford Motor Co., as well as all “five branches of the U.S. military,” the Pentagon, Justice Department, State Department, and the “Office of the President of the United States.” 

Large companies with enough resources are rebuilding their computer systems to ensure that any undetected presence of the attackers does not create future problems, but not every company has the wherewithal to do that, Williams said.

Many unknowns

The challenges of detecting and removing the hackers’ presence is complicated by how long the attackers managed to remain undetected, said Steve Grobman, the chief technology officer at McAfee, a cybersecurity company. 

Since the attack went undetected for months, it could have created “lots of opportunities [for the adversary] to go in many different directions,” Grobman said. 

“It’s like knowing a burglar has been in your house, but you don’t really know what they took, so you have to go into every room, and inventory everything of value everywhere before you have confidence of knowing what the impact was,” Grobman said. “It’s far worse in the digital environment because there are so many places for an adversary to hide.” 

For government entities whose networks may contain software code that is decades old, finding malware poses additional challenges because “people don’t necessarily know all of nuanced technical details” of the older code, Grobman said. 

All the experts said that even the most sophisticated company or government agency has little idea of the vast global network of vendors that supplies parts, components and bits of software code that go into computer systems, making it even more challenging to find which broken window the adversary may have used to gain access. 

The New York Times on Wednesday reported that a Czech tech company called JetBrains, founded by Russian engineers, may have provided the backdoor for the Kremlin to access SolarWinds, a client of JetBrains. U.S. intelligence agencies are probing the role JetBrains may have played in the hack, the Times said. 

Studying the U.S. response

Finding places on a network where hackers may have left bugs and trying to do damage control could by themselves be subject to eavesdropping by the adversaries, said Blake Moore, vice president of strategy and operations at Wickr Inc., maker of a fully encrypted messaging platform. 

If the attackers’ goal was espionage, then that could include attempts to figure out how top U.S. federal agencies and American companies respond to a hack, and trying to use that knowledge to attack with more sophistication in the future, said Moore, who previously was the deputy director of intelligence at the U.S. Cyber Command. The company’s clients, including government agencies, are using the Wickr platform to avoid exposing their moves to adversaries, he said. 

“They’re attempting to understand exactly how incident response works, who’s communicating with whom, and who is critical,” Moore said. 

It’s like being on a battlefield and watching one side mount defensive actions, trying to eject an adversary and the adversary using that knowledge to evade capture, and be one step ahead of the network defenders, Moore said. 

“They’re essentially learning how we defend ourselves and learning how our radars work, so to speak,” Moore said. “That, to me, is a huge problem.”