Email records show University of Wisconsin System cybersecurity staff raced to determine whether any of its 26 campuses or central office had been impacted by the global SolarWinds hacking incident discovered in December 2020.
According to documents, some UW institutions were running the compromised software, though it’s unclear whether attackers stole information or disrupted university networks. On December 13, 2020, the U.S. Cybersecurity & Infrastructure Security Agency issued an alert stating that computer network monitoring products made by Texas-based company SolarWinds were being “exploited by malicious actors” and that it “posed an unacceptable risk to U.S. government agencies.”
On December 16, Reuters reported the hackers were believed to be working for the Russian government and that they had been monitoring internal email traffic at the U.S. Treasury and Commerce departments for months. The story said attackers did this by hijacking an automated update process used by SolarWinds that was sent out to tens of thousands of customers. Those included federal, state, local and tribal governments along with corporations and universities.
The same day of the Reuters report, UW System Associate Vice President of Information Security Katherine Mayer sent an email to administration staff with the subject line, “Information Security Incident — Solar Winds,” according to documents obtained through a state open records request.
“It is estimated this incident was likely the result of a highly sophisticated, targeted, and manual supply chain attack by an outside nation state,” wrote Mayer. “This campaign may have begun as early as Spring 2020 and is currently ongoing. The most dangerous effects of this campaign are lateral movement within infiltrated networks and the exfiltration of data from the compromised networks.”
System IT Staff Searches For SolarWinds Vulnerabilities
In the following days, Mayer and other IT leaders within the UW System worked to identify which computer servers used by 26 campuses, the central administrative office, UW Shared Services and Extended Campus might have been using SolarWinds products. On December 18, UW System Director of Information Security Governance, Risk and Compliance Nicholas Davis emailed Mayer to ask whether the central office’s procurement staff could look through recent campus purchases of SolarWinds products.
“I’m just not comfortable with hearing, ‘We don’t have it,’ without proof of that, or even knowing what method they used to come to that determination,” said Davis.
Procurement records showed UW-Eau Claire, UW-Green Bay, UW-Oshkosh, UW-Stout and UW-Stevens Point had made purchases from SolarWinds of the past year. But that didn’t tell the whole story because the company offers multiple products, and only its Orion software was considered compromised.
The initial U.S. Cybersecurity and Infrastructure Security Agency alert included a list of instructions for Orion users. They were told to disconnect any computer system running the software from the internet, forensically image memory and operating systems for a list of filenames, and analyze records of network traffic. A subsequent alert said the SolarWinds attackers were observed bypassing the DUO multi-factor authentication program to gain access to Microsoft Outlook emails.22
On December 21, draft emails intended for chancellors, campus information security officers and the UW Board of Regents were being distributed within the UW’s central administration office. Campus officials were given instructions on how to search for signs of suspicious activity in their Microsoft email systems, while the email to regents gave a synopsis of the SolarWinds hack.
That email indicated that three UW institutions used the impacted version of SolarWinds but did not find any malicious code. Further steps were taken to disconnect the servers while IT staff uploaded a security patch provided by SolarWinds.
A large portion of the December 21 email to regents referenced two of the 15 UW institutions and included multiple lines of text and six bullet points that were redacted, aside from a mention that the “estimated number of individuals impacted / records exposed” was undetermined and that an investigation was in progress.
System President Says Institution ‘Could Be An Easy Target’
UW System interim President Tommy Thompson said he could not comment on whether the two institutions mentioned in the email were campuses or how the system was impacted. But he said hackers from around the world are constantly trying to break into UW computer systems and the SolarWinds incident was no different. Thompson likened the ongoing threat to a cybersecurity war.
“A good number of the hackers are actually employed by foreign governments and it’s like a full time job for them, in which they come to work in the morning and they’re given a computer to use to hack in across America,” said Thompson.
Thompson said part of the challenge in protecting the UW System against being hacked is its large digital footprint.
“And when you have 26 campuses and 13 universities and thousands of servers, you can well imagine that we could be an easy target,” said Thompson.
To reduce the number of ways a potential hacker can infiltrate the UW System’s computer networks, Thompson wants to consolidate them and move data to a cloud based system under an effort he is calling “IT as a Service.”
“And this is a big job to consolidate all the servers and centralize IT,” said Thompson. “So, it lessens the breadth of what can be hacked and how the hackers can get into our system.”
Other Thompson initiatives include centralizing university purchasing and administrative functions via the Procure-to-Pay automation initiative and Administrative Transformation Program. Thompson said the UW System has made good progress and he hopes to move “IT as a Service” far enough along that the next UW System president can easily finish it.
“But it’s going to be expensive,” said Thompson. “And the resources that are needed to do it are such that I’m not sure the Legislature is going to be happy with us.”
Universities Seemed Like ‘Collateral Damage’
Von Welch is the associate vice president for information security at Indiana University and executive director of a collaborative of university IT professionals called OmniSOC. He said the scope of the SolarWinds attack was huge but attackers seemed to be focused on federal agencies. Welch said other customers, like universities, seemed more like “collateral damage.”
Information security staff normally watch for hackers attempting to break in from the outside, said Welch, but this time they’d already been inside for months by the time the alert went out.
“To be honest with you,” said Welch, “I know of no organization that is sophisticated enough to be looking that hard at its major updates to catch something like that. That’s incredibly difficult. So then once you have that installed, now all of a sudden you have your attacker basically with access to you on the inside. And so then, you know, you’re playing catch up.”
Welch said the attack has reinforced the notion that anytime new software or updates are brought into a secure computer network, there’s a possibility malicious code could be hitching a ride.
“So, it’s just emphasized to us the importance of segmenting our networks, trying to keep different parts of our systems isolated from each other,” said Welch. “So, if one part of our infrastructure is compromised through supply-chain attacks like this or a phishing scheme or whatever, it doesn’t cause a huge problem throughout.”
Scott White is the program director for cybersecurity at George Washington University in Washington, D.C. He said that the U.S. Cybersecurity and Infrastructure Security Agency has provided a tool for SolarWinds customers to check for signs that attackers compromised their networks, and it appears to have worked well. But he said IT experts are still finding other vulnerabilities linked to the attack.
“There has been other malware that was transmitted, they believe, through that same backdoor,” said White. “So, that’s the issue, isn’t’ it? It’s not just the initial attack. That back door was open, and what other malware products were being distributed?”
White said he expects future revelations tied to the SolarWinds hack “for quite a while.”
Rich Kremer
