Proof of Concept: Overcoming Google Cloud Incident Response Issues with ‘Mirage’

Background

• Sygnia recently developed ‘Mirage’, Sygnia’s collection tool for Google Cloud configurations, logs and data.
• This blog is part of a series that covers investigations in Google Cloud, including infrastructure, forensic artifacts, and an introduction to Mirage.

Introduction

This blog covers two case studies of attack scenarios in Google Cloud, demonstrating how a breach in Google Workspace might lead to a compromise of Google Cloud Platform (GCP), and vice versa.

The following scenarios highlight three main ideas:

• How Mirage can be used during an investigation, specifically during the evidence collection and scoping phases.
• An exploration of example attack vectors, and full attack workflow in Google Cloud.
• General log analysis strategies, and incident response and threat hunting concepts in Google Cloud.

Case Study #1: Business Email Compromise to GCP

Scenario Context

The first scenario begins with reports about suspicious phishing emails being sent from an internal user. The ‘Joe Davis’ user account was identified responding to existing email threads with an anomalous message that lures potential victims to visit a link, under the pretense of account compromise.

Email account compromise attempts often involve hijacking existing threads to perform additional phishing operations. This technique is effective because the recipient is more likely to trust emails from a known sender, especially when the communication is part of an existing email thread.

Scoping, Access, and Initial Evidence Collection

After receiving the lead indicating a phishing attempt, our first goal is to decide what forensics artifacts should be collected, based on our understanding of the investigated network. As this scenario relates to a BEC (Business Email Compromise), we focus on collecting Google Workspace logs and Gmail configurations and data; to this end, we can leverage Sygnia’s open-source Mirage tool, which is an efficient and simple solution for evidence collection. These logs, configurations, and application data enable us to triage Joe Davis’s account activity by exploring a snapshot of existing infrastructure, and actions performed by his account.

Mirage can be installed using Google Cloud Shell with the following command:

$ python3 gcassistant.py setup --service all --organization-id 123456789012 --profile logs,configurations

After Mirage Assistant finishes the Mirage installation, a secret key that is used by Mirage Collector can be downloaded to continue with the following steps.

Once the key is downloaded, Mirage Collector can be used to acquire all available information from Google Workplace, including Gmail data, as defined in the scoping phase.

$ python3 mirage.py gw --key-file key.json --super-admin michael.strand@example all

$ python3 mirage.py gw --key-file key.json --super-admin michael.strand@example gmail --users [email protected] threads

Investigating Workspace

In the initial phase of an investigation, it is advisable to work on multiple workstreams simultaneously, to achieve maximum efficiency and respond swiftly to the threat. The three workstreams that are explained in this blog enable triaging Joe Davis’s user account activity from different angles.

The first workstream involves the identification of post-exploitation activities that were carried out within Google Workspace. Specifically, this workstream aims to uncover additional evidence related to the threat actor’s activities using the compromised account.

Our investigation starts with an examination of the Workspace logs collected by Mirage for events associated with the compromised account during the attack timeframe. During our inspection of Joe Davis activity prior to the compromise, we observed an anomaly related to the login and subsequent actions that were performed from an IPv6 address.

One of the suspicious actions performed following this anomalous login was the disabling of 2-Step Verification (2SV) – Google Cloud’s implementation of multi-factor authentication. This action was taken by the threat actor to facilitate access to Joe Davis’s account in the future, and gain persistence.

While reviewing Joe Davis’s activity, we came across another significant action involving email forwarding outside of the domain. A log entry shows that the threat actor created a new forwarding rule in Joe Davis’s Gmail mailbox, resulting in any future emails received by Joe Davis being automatically forwarded to an external email address – in this case: ‘[email protected]’. This action suggested an ongoing process of data exfiltration from Joe Davis’s mailbox, highlighting the need for enhanced monitoring and protection of sensitive data.

The following image summarizes what we discovered up until this point:

While examining the post-exploitation activities, we also investigate how Joe Davis’s account was originally compromised in a second workstream. From our experience in previous engagements, one of the most common ways to gain access to a Google account is by performing credential harvesting, through phishing emails. Mirage provides the capability to search Gmail data (email messages and threads) based on a certain timeframe, or other filters. This Mirage feature can be used to identify the initial phishing email based on multiple indicators gathered in the investigation.

When searching through the output from Mirage, we came across a suspicious message snippet of a phishing email originating from Chad Lorre, which happens to be the same email address used for the forwarding rule creation. Investigating the message further reveals the existence of a fake Google sign-in alert. This evidence shows that in all likelihood, the threat actor combined an MFA push attack with a phishing campaign in order to gain initial access. During such an incident, the details can be confirmed with the relevant user.

Summarizing the second workstream provides a wider view of the actions performed by the threat actor in Google Workspace:

While continuing to search for malicious activities in our two workstreams, we can also work on a third workstream involving GCP.

Investigating Google Cloud Platform

As GCP and Google Workspace share the same identity management solution, our next step is to confirm whether Joe Davis was granted access to GCP resources, and if any unauthorized activities have been detected. To achieve this, we can interview relevant personnel within the company and analyze GCP configuration data, which we gathered using Mirage. We also investigate GCP as the third workstream, in parallel.

Mirage provides the ability to collect configuration data across specific projects or the entire GCP organization, as can be seen in the following command:

$ python3 mirage.py gcp --key-file creds.json configurations --organization-id 123456789012 --configs all

By using the IAM role bindings section in the Mirage output, it is possible to correlate an actor, affected resource, and roles and permissions. After filtering the data, we see that the Joe Davis user account holds two roles: Editor and Project IAM Admin over a specific project. By exploring the information stored in the resource hierarchy, which is another part of Mirage output, we can determine that Joe Davis was given access to a project called ‘unit-testing-368207’.

Following this discovery, we leverage Mirage to retrieve logs related to the ‘unit-testing-368207’ project using a single command:

$ python3 mirage.py gcp –key-file creds.json logs –logs all_logs –project-id unit-testing-368207 –start-time 2022-10-01T00:00:00Z –end-time 2022-12-01T00:00:00Z

After collecting the relevant GCP logs, we investigate further by filtering Joe Davis’s user account activity within the specified timeframe of malicious activity.

An examination of these logs reveals evidence of virtual machine creation through the ‘beta.compute.instances.insert’, as shown in the methodName column. Closer inspection of the log event shows that the newly-created VM was named ‘instance-1’, and was built with powerful and expensive computing hardware. As the reason for virtual machine creation is still unclear at this point, it is necessary to narrow down our search by filtering for activities associated with the instance name within the known timeline.

When investigating the logs associated with the instance, we come across events indicating startup script execution. In GCP, a ‘startup script’ is a file associated with a virtual machine (VM) instance that performs actions during its startup process. The script can be used to perform various tasks, such as installing software, updating configurations, or setting environment variables. In this case, the threat actor used the startup script to build ‘XMRig’ from source code. Further research reveals that one of the significant impacts of the cyberattack is the execution of crypto mining software.

Continuing with our investigation, we return to the GCP logs and delve more deeply into subsequent actions that were performed by the Joe Davis account. Following the instance creation, the next actions appear to be related to a newly-created service account called ‘qa-audit-manager’, within the ‘unit-testing-368207’ project.

In GCP, service accounts are used to provide non-user Google identities with access to resources in GCP without the need for traditional login. Unlike user accounts, service accounts do not have passwords, and instead use RSA key pairs for authentication. The service account creation was created by the threat actor as a persistence method. To track the actions performed by the ‘qa-audit-manager’ service account that was established by the threat actor, we use the key ID associated with the RSA key pair. This enables us to monitor and detect any further malicious activities performed using the service account.

Following the creation of the new ‘qa-audit-manager’ service account and a key pair, we observe a ‘SetIAMPolicy’ action performed by Joe Davis, which grants the Editor role to the service account for the ‘unit-testing-368207’ project. The Editor role enables the creation, modification, and deletion of almost all resources within GCP. This evidence leads us to conclude that the threat actor established persistence by giving the service account almost unlimited access to resources within a single GCP project.

Summary

Our investigation into this scenario was initially triggered following reports of anomalous phishing activity from Joe Davis’s account. We used Sygnia’s open-source Mirage tool to collect log events from Google Workspace, in order to identify the activities conducted by the compromised account. We also used Gmail thread data, which was collected by Mirage, to understand how the account was compromised. Using Google Cloud service links and information collected from company personnel, we utilized GCP configurations and logs to pivot and detect virtual machine creation for crypto mining and service account persistence. Throughout these investigation steps, we used a combination of Mirage, our understanding of Google Cloud infrastructure, and forensic evidence to get a comprehensive understanding of the incident, and to react in an efficient and methodical way.

Now, let’s shift our focus to another scenario that demonstrates a different approach to compromise in Google Cloud. While our first investigation started with a phishing alert in Google Workspace, this new case study begins with a different set of circumstances.

Case Study #2: Exposed Service Account Compromised to Workspace

Scenario Context

The second scenario begins with a ransom email being sent to an organization’s Google Cloud administrators. In the email, the threat actor claims to have taken “sensitive employee data”.

The investigation starts by identifying where the data was stolen from, followed by remediation actions to minimize further damage. Speaking with the  organization’s Google Cloud administrators and other relevant personnel, reveals that the leaked information is stored in a specific database hosted in GCP. With this in mind, we can start analyzing the activities involving this database.

Scoping, Access, and Initial Evidence Collection

Accessing the investigated environment is done in a similar manner to the installation, access, and setup described in the first scenario. It is important to mention that the installation command includes both GCP and Google Workspace, as they share infrastructure, such as identity management. However, if required, the access setup using Mirage Assistant can be done separately on GCP, or on Google Workspace.

After setting up the required access, we can collect relevant GCP logs for projects and folders containing information that could be related to the stolen data. We also utilize Mirage to collect the configurations of the entire organization in the GCPenvironment.

$ python3 mirage.py gcp --key-file creds.json configurations --organization-id 234567890123 --configs all

$ python3 mirage.py gcp --key-file creds.json logs --logs all_logs --project-id budget-368114,state-audit-reports,mergers-and-acquisitions --folder-id 81778926867,994323795756 --start-time 2023-01-09T00:00:00Z --end-time 2023-02-09T00:00:00Z

Investigating Google Cloud Platform

After collecting the relevant logs and configuration data from GCP, we filter the results to find events associated with the Cloud SQL service, to look for anomalous activities. Cloud SQL is GCP’s built-in solution for hosting databases. Anomaly analysis can consist of sensitive API method calls (e.g., database modification events), or of unique events based on several parameters.

Our examination of the Cloud SQL events reveals suspicious activities that indicate the involvement of a service account. The caller IP and API method calls are irregular, suggesting that the service account was used by the threat actor.

Identifying that a suspicious service account was used provides us with a new lead to discover additional opportunities for pivoting, and advancing the investigation.

As in the first scenario, the incident response team manages multiple workstreams simultaneously, to achieve maximum efficiency and to provide a swift response to the threat. In the first workstream, we focus on identifying post-exploitation activities in Google Cloud Platform.

By inspecting prior events, we discover the presence of multiple API methods for listing and getting certain GCP resources.

All of these actions were performed within a matter of minutes in the context of multiple Google services. This behavior indicates GCP service enumeration activity – more specifically, the discovery phase of an attack.

Additional actions made by the same service account include an update to the root user of the SQL instance, which contained the stolen data. The update to the root database account suggests that its password was reset in order for the threat actor to gain access.

Following the root database account update, the compromised service account was observed using the ‘cloudsql.instances.update’ method. The threat actor used this API method to change the ACL (Access Control List) of the SQL instance, to allow communication from ‘any IP address’ (0.0.0.0/0), effectively making the database public. These actions provided the threat actor with access to information contained in the database, ultimately facilitating data exfiltration, and the subsequent ransom threat.

The following diagram summarizes this first workstream:

As we shift context to the second workstream, the insights we acquired about the incident up to this point help us to identify how the threat actor gained initial access. By analyzing logs, we ascertain that a service account was used by the threat actor, and that the associated events were conducted after authenticating using the same single access key.

To trace the usage of this key, we can discuss with relevant company personnel how the service account key legitimately authenticates applications and other services. After additional discussions and investigation, we are directed to public code repositories associated with the service account. Using code scanning tools, we are able to identify that the service account key was exposed in a public code repository.

By investigating information captured in logs and leveraging cloud security concepts, we discover how the threat actor gained initial access to the environment.

After investigating the GCP post-exploitation activities and determining how initial access occurred, we can begin to explore potential links between the identified service account and Google Workspace. As mentioned earlier, Google Workspace and GCP share the same identity management system, which allows for shared access to resources in both systems.

Investigating Google Workspace

At this stage, we use Mirage to continue the evidence collection, this time from Google Workspace. As was demonstrated before, Mirage provides a quick and efficient method of gathering both configuration and log data from Google Workspace:

$ python3 mirage.py gw --key-file creds.json --super-admin [email protected] admin_directory all

$ python3 mirage.py gw --key-file creds.json --super-admin [email protected] logs –logs all_logs --user all_users

In combination with the collected logs, we can leverage our knowledge of how service accounts access data across Google Workspace to detect abuse. A service account with delegated authority can impersonate any user while behaving as an application. This identity impersonation in google Cloud is called domain-wide delegation (DWD). When DWD is setup for a service account, the service account is authorized to act on behalf of any user for relevant given access scopes (permissions). Later, whenever an “authorize” event is registered in the logs, it means that a specific user authorized access to an application, or a service account, to act on its behalf. Therefore, rare authorize events that involve our compromised service account could lead us to potential exploitation.

During the investigation, we identify a rare authorize event associated with the ‘Ryan McDonald’ Super Admin account, which performed anomalous events after being authorized with six access scopes. Using the client ID captured in the log enables us to track the application responsible, which turns out to be the compromised service account.

The fact that a service account was authorized with access scopes by a user account suggests that DWD was already established. Although there is no particular session ID or field that tracks events initiated by DWD, we can still identify them by examining the actions that follow the ‘authorize’ event, which are linked by IP address, actor, and time. During the investigation, we observe the compromised service account utilizing Super Admin privileges to create a new user called ‘Reports Admin’, and subsequently providing it with administrative privileges. As a result, this new account obtained complete control over all aspects of Google Workspace for all users. These actions enabled the threat actor to create persistence in the Google Cloud environment.

After the ‘Reports Admin’ account logged in, it modified a system-defined rule related to the notification settings for the Google Alert Center. Specifically, it disabled the ‘Suspicious Login’ alert, as a defense evasion technique that can help to hide future logins by the threat actor.

Summary

In this scenario, the threat actor’s ransom note led to the discovery that information had been stolen from a GCP-hosted database. We used Mirage to gather log and configuration data, which helped to identify malicious actions that were conducted from a compromised service account associated with the database. Further investigation revealed public repositories exposing the credentials that were used for initial access. Using our knowledge of service account access and domain-wide delegation, we analyzed evidence in Google Workspace to detect additional persistence and defense evasion activity. This scenario highlights lateral movement from GCP to Google Workspace, in contrast to the first scenario, which involved lateral movement from Google Workspace to GCP.

Conclusion

The purpose of this blog – along with Sygnia’s previous blogs and the release of our open-source Mirage tool – is to assist organizations in overcoming incident response challenges in Google Cloud. In this blog, we presented two real-life compromise scenarios in Google cloud to highlight several key incident response concepts:

1. How comprehensive Google Cloud cyber-attack look through configuration, logs, and data.
2. How Sygnia’s Mirage tool enabled us to gather the relevant evidence efficiently and easily.
3. How threat actors can leverage existing access controls to further their malicious objectives.
4. How the connection between GCP and Workspace can be utilized by threat actors to perform lateral movement in both ways.

We extend an invitation to the cybersecurity community to actively engage by sharing insights on the tools, raising questions, and contributing to the development of Mirage. Stay tuned for upcoming Google Cloud blog posts.

Contributors: Itay Shohat, Wesley Guerra, Shani Adir

Google Cloud forensics blog series

• Incident Response in Google Cloud: Foundations
• Incident Response in Google Cloud: Forensic Artifacts
• Announcing ‘Mirage’ – New Opensource Tool to Combat Google Cloud Incident Response Challenges
• Proof of Concept: Overcoming Google Cloud Incident Response Issues with ‘Mirage’

This advisory and any information or recommendation contained herein has been prepared for general informational purposes and is not intended to be used as a substitute for professional consultation on facts and circumstances specific to any entity. While we have made attempts to ensure the information contained herein has been obtained from reliable sources and to perform rigorous analysis, this advisory is based on initial rapid study, and needs to be treated accordingly. Sygnia is not responsible for any errors or omissions, or for the results obtained from the use of this Advisory. This Advisory is provided on an as-is basis, and without warranties of any kind.