Cloudflare has revealed its systems were compromised on Thanksgiving last year, leading to source code being accessed by threat actors.
The IT service provider believes the attack, which took place on November 23, 2023, was perpetrated by a nation-state actor, who used credentials stolen during a breach of identity and access management (IAM) specialist Okta.
Cloudflare admitted that it “failed to rotate” its credentials that were stolen during the Okta breach.
No customer data or systems were affected during the incident, which Cloudflare attributed to its zero trust environment limiting the threat actor’s ability to move laterally.
The attack was stopped on November 24, with all threat actor’s access and connections terminated.
Following an independent analysis by Crowdstrike, Cloudflare provided details of the incident in a blog published on February 1, 2024.
How Were Cloudflare’s Systems Compromised?
During the Okta breach on October 18, 2023, the attackers stole one service token and three service account credentials belonging to Cloudflare.
These provided the following access to the cloud provider’s systems:
- A Moveworks service token granted remote access into the firm’s Atlassian system
- A service account used by the SaaS-based Smartsheet application, which gave administrative access to Cloudflare’s Atlassian Jira instance
- A Bitbucket service account allowing access to the source code management system
- An AWS environment that did not have access to the global network or contained customer or sensitive data
These credentials were not rotated because “mistakenly it was believed they were unused,” Cloudflare said.
The threat actor began searching for ways to access Cloudflare’s systems on November 14 using the stolen credentials. On November 15, it successfully accessed Atlassian Jira and Confluence using the Moveworks service token.
The Smartsheet service account was then used to access the Atlassian suite. Once in these systems, the attackers searched for details about the configuration and management of Cloudflare’s global network, accessing various Jira tickets.
The threat actor also created an Atlassian user account via the Smartsheet credential. This was to ensure they’d have persistent access to the Atlassian environment should the Smartsheet account be removed.
Following a few days “break” from accessing Cloudflare’s systems, the attacker gained continuous access to the Atlassian server on November 22 after installing the Silver Adversary Emulation Framework tool, enabling command and control.
Over the next day, the threat actor viewed 120 code repositories and downloaded 76 of them to the Atlassian server.
These downloaded repositories were almost all related to system configuration and management at Cloudflare, such as how identity works and remote access. This is likely to find ways to mount a subsequent attack on the provider.
The company has treated the 76 downloaded source code repositories as exfiltrated by the attackers.
Cloudflare Detection and Remediation
The threat actor was detected at 16.00 UTC on November 23, leading to Cloudflare’s security team deactivating the Smartsheet service account 35 minutes later.
The user account created by the attacker was then discovered and deactivated 48 minutes later, at 17.23.
The Sliver tool was removed at 11.59 on November 24, and the last known threat actor activity was at 10.44 on the same day.
Cloudflare revealed that the threat actor attempted to access a “myriad” of other systems on its network, but its presence was limited to the Atlassian suite. This included an attempt to access a data center that Cloudflare had not yet put into production in São Paulo, Brazil.
This meant no customer data or systems were accessed.
The firm said that this failure to move laterally was due to its zero trust architecture, which enforced access controls, firewall rules and use of hard security keys.
“We are confident that between our investigation and CrowdStrike’s, we fully understand the threat actor’s actions and that they were limited to the systems on which we saw their activity,” wrote Cloudflare.
Following the incident, Cloudflare instituted a project dubbed “Code Red” to harden all controls in its environment and secure against future intrusion. This included analyzing the 76 stolen source code repositories to remediate embedded secrets, vulnerabilities and other ways in which an attacker could use them to mount a subsequent attack.
To prevent the attackers finding a new way back in, the firm undertook a “comprehensive” remediation effort, including:
- More than 5000 individual credentials were rotated
- All test and staging systems were physically rotated
- Forensic triages were performed on 4893 systems
- Every machine in its global network, including all Atlassian products, were reimagined and rebooted
Suspected Nation-State Intrusion
The sophisticated and methodical nature of the attack suggests the perpetrator was a nation-state attacker, the firm added.
“Based on our collaboration with colleagues in the industry and government, we believe that this attack was performed by a nation state attacker with the goal of obtaining persistent and widespread access to Cloudflare’s global network,” wrote Cloudflare.