CircleCI, a well-known CI/CI (continuous integration and continuous delivery) platform provider, fell victim to an advanced Cyber Attack and was alerted to a suspicious GitHub OAuth activity by one of CircleCI’s customers on December 29, 2022. The attacker planted malware in the laptop belonging to a CircleCI’s employee and gained unauthorized access to its production systems to extract sensitive data. Although the data was encrypted, the attacker managed to obtain the encryption keys, which could potentially grant them access to the decrypted data. The CircleCI’s breach was significant as CircleCI was serving prominent companies such as Meta, Okta, Salesforce, and Airbnb. In this article, we analyze the details of the incident, how it happened, and what measures can be put in place to protect against similar attacks.
What happened in the CircleCI’s breach?
The attack started with a compromise of a CircleCI employee’s laptop through malware. The attacker could access the employee’s laptop and stay undetected, which allowed them to gain access to the company’s network. The attacker reused the existing login session found in the employee’s laptop to impersonate the employee and gain further access, effectively allowing the attacker to by-pass two-factor authentication. This allowed them to move further laterally within the network to gain access to production level systems and data.
The key sequence of steps in the attack was:
- The employee laptop was compromised on December 16, 2022
- Attacker performed reconnaissance on December 19, 2022
- Attacker gained access and collected data on December 22, 2022
How the CircleCI’s attack succeeded
The attack succeeded due to a failure of several essential controls:
- Firstly, the attacker compromised a laptop belonging to CircleCI’s engineer and planted a backdoor. CircleCIs’ anti-malware solution did not detect the malware, which allowed the attacker to further continue their activities and remain undetected for an extended period (step 1).
- Secondly, the attacker reused the web session cookie stored on the laptop (step 1).
- Thirdly, this attacker impersonated a CircleCI employee who has been authorized via multi-factor authentication and gained access to production systems (step 1).
- Fourth, the attacker further escalated their breach by successfully downloading an array of highly sensitive data including SSH keys, API tokens, OAuth tokens, and an AWS IAM access key (step 1).
- Fifth, armed with the SSH keys and tokens, the attacker could seamlessly reuse these credentials to infiltrate not only CircleCI’s internal systems but also gain illicit access to invaluable resources such as the customer’s repo and AWS resources (step 2).
- Finally, the attacker extracted encryption keys from CircleCI’s customer code repository. Despite the company following best practices by encrypting sensitive information such as AWS keys and GitHub tokens, the attacker gained access to the keys needed for decryption (step 2).
As is obvious the attack was multifaceted, with attacker abusing the trust present in the employee’s laptop to impersonate authorized requests. Their careful network reconnaissance allowed them to plan further attacks, increasing their chances of success. This allowed the second phase of the attack, i.e., the data exfiltration attempt to succeed, allowing them to compromise highly sensitive data.
What does CircleCI do?
CircleCI took several steps to contain the breach once it was detected to limit the blast radius of the attack. The level of exposure post-attack was challenging to determine as the compromised staff had production access and access to customer tokens and keys. The attacker might have exfiltrated further data without leaving any traceable evidence. CircleCI took a transparent approach to the incident and informed its customers of how the attack had taken place. It issued regular updates to customers and advised them to rotate all of their credentials, such as SSH keys, OAuth tokens, etc., to mitigate the risk of further misuse by the attacker. It also revoked Project API tokens and personal API tokens to limit the potential entry points the attacker could exploit following the attack.
Secondly, it recognized the security failings that allowed the attack to succeed and initiated a comprehensive review of its environment. It strengthened production access controls, introducing additional controls for employees needing access to systems. This was intended to mitigate the risk of session compromise in the future via stolen session cookies.
Lastly, it enhanced its capabilities to detect specific behaviors indicative of attacks such as the one that occurred. Attacks such as lateral movement and malware activity were focused on to ensure no similar attacks occurred.
CircleCI also assured its customers of regular security reviews and risk assessments to identify weak areas and areas of improvement. There have been reports of customers who have reported attacker misusing the stolen credentials, so it is possible that the impact of the breach is yet to be determined.
Key lessons to learn from the CircleCI breach
The CircleCI data breach is a prime example of how sophisticated attacks can undermine even the most reliable security controls such as encryption and multi-factor authentication. Advanced threats are aware of these controls and adopt techniques to evade or bypass them. To remain updated, continually evaluating your security posture against the latest threats is essential.
Some of the key lessons from this breach are:
- Traditional security controls may no longer be enough. Companies need to adopt techniques like biometrics and machine-learning-based anomaly detection that can detect subtle variations in behavior that human security analysts might miss.
- Devices like laptops and smartphones remain a vulnerable entry point into a network as it was one compromised endpoint that allowed the attacker the access they needed to carry out further attacks. Companies need to look into re-architecting their networks based on Zero Trust principles. This architecture assumes that every request is potentially malicious and no implicit trust is assumed regardless of location or device.
- Key Management best practices remain as critical as ever. Even though CircleCI had implemented encryption, the attacker could retrieve the encryption keys from a running process on the machine. Protecting encryption keys is crucial to key management, as a compromised key can undermine the entire security strategy.
On a positive note, it should be mentioned that CircleCI displayed proper transparency regarding the attack and took immediate remedial action once they detected it. Despite the security weaknesses that allowed the attack to occur, companies should note how they informed customers and released notifications regarding the scope of the incident.
Conclusion on the CircleCI breach
Like SolarWinds, the CircleCI attack is a wake-up call for companies to not be over-reliant on controls like multi-factor authentication and encryption. Instead, a company’s cybersecurity posture must be evaluated continuously to assess its resilience against advanced cyber-attacks. By adopting controls like zero-trust architecture, machine learning based detection, and a proactive stance towards cybersecurity; companies can significantly mitigate the risk of falling victim to such attacks like CircleCI.
How InsiderSecurity can help
InsiderSecurity provides robust solutions to detect and thwart attacker tactics and techniques used in the CircleCI breach. With InsiderSecurity’s solutions, organizations can detect potential attacks early, significantly reducing or even preventing data loss.
For example, InsiderSecurity’s Automated UEBA (User and Entity Behavior Analytics) solution effectively counters the risk of compromised servers. It identifies the unauthorized use of stolen SSH keys, which is a tactic used by the CircleCI attackers. Additionally, InsiderSecurity’s CSX cloud security solution secures cloud environments against unauthorised access. It expertly identifies suspicious activities, such as the misuse of stolen session cookies or the unauthorized use of cloud credentials for platforms like AWS and other SaaS services — tactics similar to those employed by the CircleCI attackers. These comprehensive measures ensure that your digital assets remain secure against evolving cyber threats.
