SAN ANTONIO — A ransomware group identified by Rackspace Technology Inc. as “Play” used a new method to penetrate the cloud computing company’s hosted Microsoft Exchange network, leaving its customers without access to their email, contacts and calendars.
It’s still not certain that all of them will regain full access to their hijacked data.
In interviews, company executives and outside advisers said an internal investigation into the ransomware attack blamed for the shutdown identified the group and found that it used a Rackspace customer’s credentials for an email account to gain access to a company server Nov. 29. On Dec. 2, the ransomware attack was deployed.
The exploit used to gain access to the network and steal data was previously unknown, Rackspace said.
Its customers began having difficulty accessing their accounts that Friday in early December, leading the company to shut down its system. It disclosed the attack four days later, on Dec. 6.
On ExpressNews.com: Damage control: Rackspace criticized by customers, experts for its PR response to ransomware attack
Rackspace said it does not know how the hackers obtained the credentials. The company declined to identify the compromised customer or disclose whether it paid a ransom to regain access to customers’ data.
While its internal investigation is complete, Rackspace said the FBI is continuing to investigate and that the company is cooperating with the agency. The FBI has declined to confirm or deny that it is investigating.
Amid uncertainty about the attack and concerns about the company’s response, declines in the company’s stock price have accelerated, pushing it to a 52-week low Tuesday. Rackspace shares have plummeted more than 80 percent since January, with about half of that decline since the attack.
Chief Product Officer Josh Prewitt said the company did not share more detailed information about the attack sooner because it wanted to maintain credibility.
“A lot of people speculated around root cause, and they got it wrong,” he said. “We didn’t want to do that because our reputation and our credibility — and maintaining our customers’ trust — are all really important to us. Now that the investigation has concluded, we’re confident that we know what happened and wanted to be transparent and forthcoming about root cause in an effort to lift up the entire security community.”
Skipped updates
The speculation he referred to began as word of the outage spread through San Antonio’s tech community Dec. 2. Many were questioning whether Rackspace had properly patched known vulnerabilities in its hosted Exchange system — essentially updating software and operating systems to close gaps in security.
As Rackspace told it, Microsoft on Nov. 8 released 66 security patches in what it called its November 2022 Patch Tuesday event. Those included fixes for high-severity vulnerabilities in Exchange known as ProxyNotShell. Patches are software and operating system updates that address security vulnerabilities in a program or product.
Before the patches were released, Rackspace in late September used URL blocking to mitigate potential known vulnerabilities, it said. But Rackspace did not implement the new patches in November because, it said, there were reported operational issues with them that came out that Tuesday.
On ExpressNews.com: Rackspace says customers will start getting access to ransomware-hit data within days
“The company thought it was mitigated, thought it had done what it was supposed to do to mitigate the risk that was disclosed at that time,” an outside adviser to the company said on the condition of anonymity. “Unbeknownst to us, and everybody, there was another risk associated with a patch that same day.”
CrowdStrike, an Austin-based cybersecurity company hired by Rackspace to conduct the internal investigation, has reported that it discovered the new exploit method, which it dubbed “OWASSRF.”
CrowdStrike, which did not name Rackspace in its report issued Dec. 20, said the new exploit method consisted of a vulnerability to achieve “remote code execution” through Outlook.
CrowdStrike said it investigated several ransomware attacks by the Play ransomware operation in which the entry points were suspected to be Exchange ProxyNotShell vulnerabilities. But CrowdStrike said it found no evidence of exploitation of the ProxyNotShell vulnerabilities and instead learned that the hackers used the credentials to gain access to the Outlook server.
The method, the company said, is “a previously undisclosed exploit method for Exchange.”
Rackspace Chief Security Officer Karen O’Reilly-Smith said there had been widespread speculation that the root cause of the attack was the result of a ProxyNotShell exploit.
“We can now definitively state that is not accurate,” she said. The company is “now highly confident” that the root cause in the case involves a “zero-day exploit,” she said, using a term meaning a type of attack previously unknown.
Recovery progress
Rackspace said Wednesday that nearly 45 percent of affected customers have regained “access to some or all of their data, and that number continues to climb each day.”
In its latest status update Tuesday morning, the company said its data recovery process for its hosted Exchange email customers was “progressing as expected.” But, it noted: “Due to the nature of the incident, certain elements of email and other data may remain unavailable to customers.”
On ExpressNews.com: Rackspace’s reputation taking a hit as response to ransomware attack falls short of customers’ hopes
Some customers have access to Exchange data from before the ransomware attack. But only those customers who migrated to Microsoft 365 or another new email platform have access to email data since the attack.
Also, customers who set up email forwarding can’t access email data from before the attack through the data recovery process. Instead, they are being directed to the archives of the forwarding address.
“Our internal and external cybersecurity experts have and continue to work diligently to streamline the data recovery process through the dedicated data recovery workstream,” the company said on its website.
Last week, Rackspace said it was continuing to “make progress” on restoring Exchange customers’ access to data. By the end of the week, it said it had recovered more than 50 percent of affected emails for a number of customers. The company did not disclose how many customers could begin downloading their emails.
“We are focused on safely extracting and delivering recovered data to our customers in an organized and secure manner and have been following an extensive and systematic process to do just that,” the company had said. “After repeatedly testing this process to be sure it goes as smoothly as possible, delivery of recovered data to Hosted Exchange customers is now underway.”
On Dec. 16, two weeks after the attack, Rackspace said it was ready to begin restoring data, which many customers have said is their biggest concern and analysts have said will be an indicator of whether Rackspace’s response to the attack is seen as a success.
“We have a very high degree of confidence that the vast majority of customers are going to be able to get their data back,” Prewitt said in an interview then.
Rackspace shares gained 30 cents, or more than 11 percent, to close Thursday at $2.90.
eric.killelea@express-news.net