Disaster Recovery and the SolarWinds Exploit

One of the biggest malware attacks in recent history was the SolarWinds exploit where malware was spread through SolarWinds’ software update process to thousands of government and private sector customers. SolarWinds is a well-known provider of IT management products and their Orion infrastructure monitoring and management platform was the product that was attacked. Automatic updates are a critical component for almost all software these days and this cyberattack compromised SolarWinds’ software update service for the Orion product.

This attack executed the Sunburst malware that was planted in the Orion management platform. The Sunspot exploit that the attackers used to insert the Sunburst malware into Orion updates was later isolated by cybersecurity firm CrowdStrike. The Sunburst malware would go dormant for a period of time to avoid detection. Then it would begin fetching additional instructions from its command-and-control (C2) server. These instructions enabled it to transfer files, execute new commands, monitor the system, and control systems. As the different organizations downloaded and installed Orion updates, they also unknowingly installed the malware on their own systems. The FBI, Office of the Director of National Intelligence, and the Cybersecurity and Infrastructure Agency in a joint statement claimed that Russia was the likely culprit behind the attack but the Russian government has denied involvement.

The malware was spread to many government departments including Commerce, Treasury, Homeland Security, and Justice. In addition, the cyberattack hit as many as 18,000 business users of the Orion software management product. The cyberattack initially occurred sometime between March and June of 2020 and wasn’t discovered until early December. Following up on their own security breach, the cybersecurity research firm FireEye discovered the SolarWinds breach and reported it to the U.S. National Security Agency (NSA), who it also turns out was a SolarWinds customer.

The Road to Recovery

There’s no doubt that this attack can be classified as a disaster for those organizations that it hit and that some level of disaster recovery procedures are required to recover from it. Finding and eliminating the malware installed is likely to be a costly affair. Estimates have put the cost of recovery upward of $100 billion over the course of several months.

The Cybersecurity & Infrastructure Security Agency (CISA) directed federal agencies to remove and disable certain SolarWinds products and start hunting for additional malware on their networks. Unfortunately, simply installing a new version of the Orion software would not be sufficient to correct the security breach. The length of time between the time when the attack was perpetrated and when it was discovered makes recovery difficult. First, the lengthy discovery period makes it difficult to simply rollback to a previous state-in-time. You can’t simply restore all your systems to a state of three months ago. There could be the potential for too much data loss. Plus, other ongoing updates of multiple related components could result in application errors. The lengthy period between when the malware gained a foothold in the victims’ networks and the time it was discovered also complicates remediation and eradication. In that time the malware could have performed numerous actions on the infected infrastructure including the creation of additional bogus credentials and the installation of additional malware components across the network.

To prevent further attacks SolarWinds is deploying more robust threat protection and detection tools. They have also reset the credentials for all users in both their corporate and development domains and have begun enforcing multifactor authentication (MFA) across the board. SolarWinds have also hired a number of high profile cybersecurity experts including the former director of the Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency, Chris Krebs, former Facebook security chief Alex Stamos and a forensics expertise from CrowdStrike.

For organizations that were affected by this breach, the first step would be to get the latest updates of the Orion software that have been sanitized. Next, identify all of the systems that were impacted as well as those systems that interacted with them. Evaluate your DR plans to see if there are contingencies for the impacted systems and if there are consider enacting them. Finally, it would be prudent to rebuild all of the systems that were affected as well as any systems that were connected to them in order to ensure that any undetected presence of the malware is actually gone.