Newsletters
Risk Central
Digital EditionSolarWinds CEO Explains What It’s Like to Inherit a Massive Cyber Attack
While enjoying his birthday dinner, Sudhakar Ramakrishna got a troubling phone call: SolarWinds had suffered a serious security breach.
Ramakrishna struck a deal to become the company’s CEO just days earlier. It would be weeks until he officially started his new job, but there was no time to waste.
Hackers appear to have used SolarWinds as a springboard to potentially gain access to 18,000 organizations, including Fortune 500 companies like Microsoft and government agencies like the U.S. Treasury Department, the Cybersecurity and Infrastructure Agency (CISA), the Department of Homeland Security (DHS), and the U.S. Department of State, and the U.S. Department of Energy (DOE).
Microsoft CEO Brad Smith called it the “largest and most sophisticated attack the world has ever seen” and estimated that more than 1,000 engineers worked on the project. A ZDNet columnist even called it “the Pearl Harbor of American IT.”
On April 15, the United States “formally named Russia’s Foreign Intelligence Service as the perpetrator of the SolarWinds cyber attack” and issued sanctions against the nation, according to NBC News.
Transparency Equals Protection
Ramakrishna discussed the attack in incredible detail during a Marsh webcast called “SolarWinds Cyberattack: Lessons Learned.” During the webcast, he explained why he believes SolarWinds was attacked and described how the company has since been responding.
Sudhakar Ramakrishna, president and CEO, SolarWinds
It wouldn’t be surprising if Ramakrishna and the SolarWinds team were tight-lipped about the attack; the damage appears devastating and an investigation is ongoing. But Ramakrishna says collaboration and transparency are crucial to cyber security.
“In the security software industry, an unwritten rule is that we’re all protecting one another,” he said.
That collaboration helps companies inform affected people immediately and apply remediation efforts as quickly as possible.
“We felt it was our responsibility to ensure that whatever we learn out of this, we project so that others can look at their supply chains and do the best they can to protect themselves,” he continued.
What Exactly Happened?
Hackers injected malicious code into a software update sent by SolarWinds to its customers. The hackers got in during a billing process that turns the computer code from computer programming language into binary numbers — 1s and 0s — then adds the SolarWinds stamp of approval. Ramakrishna said that the process takes approximately “a millisecond.”
“There was no way for a developer or the development team to detect it, because it happened during an automated process and it was a very short window of time. In essence, they were hiding in plain sight,” said Ramakrishna.
Attacking a supply chain was nothing new, but this particular attack “had not been seen before, and the stealthiness and patience with which it was done made it even more difficult to detect,” he said.
The Price of Ubiquity
Why would a nation state attack SolarWinds?
“The best answer I can give you is that this is the price we pay for being ubiquitous. We have more than 300,000 customers,” said Ramakrishna. “If I’m trying to create a supply chain attack, my goal would be to get it as broad a footprint as possible.”
The attack potentially exposed data like user IDs, passwords, financial records and source code. Naturally, customers began wondering if they were exposed — and if so, could they take action. Jason Bliss, chief administrative officer and general counsel of SolarWinds, made sure his team proactively sent patches to any customers who appeared likely to have suffered damage.
“The good news is they were able to quickly develop cleanse code and patches,” said Bliss, who also joined the Marsh webcast. He said it’s very possible that the attack was targeted, meaning the final tally of affected customers could be lower than expected.
SolarWinds software runs with a high level of privileges inside customer accounts. Once an attacker invaded, it had access to a vast amount of information. In the wake of the attack, SolarWinds is attempting to perform the same functions inside customer accounts but without the high level of privilege. SolarWinds is also adding redundancies into its billing systems so hackers would be forced to launch three simultaneous attacks — in the same manner, at the same time.
“By shifting the characteristics of the bill systems, we are keeping them guessing as well,” said Ramakrishna.
Another early remediation step was working with Marsh and its insurance carriers to understand the scope of services and assistance and to help with claims.
Secure by Design
Security in software design is often developed in later stages. Design a product, get it to work, then figure out how to keep it secure. Ramakrishna said that process needed rethinking.
“We have to inject it right at the design phase,” he said, noting that the company is now using the phrase “secure by design” as a defacto ethos for safety. It represents a change in company culture at SolarWinds. Ramakrishna said the company is working to create a “security consciousness” among everyone at the organization — not just engineers or software developers.
“We are living security across the organization. That starts with me and permeates across the organization,” he said.
The company has also given its security team autonomy to hack the company from within. They also have the ability to stop a piece of code from being delivered if they detect a potential problem — without seeking approval from higher-ups.
The SolarWinds incident may be one of the largest cyber attacks in history, but it certainly won’t be the last. That’s why Ramakrishna is stressing communication and collaboration.
“We know that more and more of these things are going to happen,” said Ramakrishna. “The only way we can avert them is through learning and sharing — and doing so constantly.”
“The uniqueness of the situation doesn’t fit a mold,” said Bliss. “Marsh and the carrier helped our customers get an understanding of what happened, educated them on the big picture, and made the claims process fairly straightforward and helpful.” &
