Announcing ‘Mirage’ – New Opensource Tool to Combat Google Cloud Incident Response Challenges

Background

• Today Sygnia released ‘Mirage’, an opensource tool designed to streamline Google Cloud forensic evidence collection. Together with the information in Sygnia’s ‘Incident Response in Google Cloud’ blog series, the Mirage tool can help organizations overcome these challenges.
• The increased use of Google Cloud and of cloud-related incidents has highlighted a lack of research and documentation available to security teams during investigations.
• There are three major challenges associated with conducting incident response, threat hunting, and posture assessments in Google Cloud: Knowledge, Access and Collection, and Effective Analysis.
• This blog is part of the Google Cloud blog series published by Sygnia with the intention of sharing accessible knowledge about Google Cloud forensics and easing investigators’ work.

Introduction

In this blog Sygnia introduces the ‘Mirage’ opensource tool, and demonstrate how it can be used to simplify incident response activities, and improve an organization’s security posture. This blog shows how Mirage can be used to streamline environment access and evidence collection in investigations that involve both Google Workspace and GCP.

Google is known as a challenging environment for incident response teams; Sygnia published two blogs that serve as an introduction to incident response and threat hunting in Google Cloud – which includes Google Workspace and Google Cloud Platform (GCP). The first – Google Cloud Infrastructure – introduced readers to the Google Cloud ecosystem, outlining how Google Workspace and GCP work together, and share the same identity management infrastructure. The second – Google Cloud Forensic Artifacts – explored the forensic artifacts that investigators can use to handle various types of activities, such as incident response, threat hunting, and posture assessments.

This blog serves not only as an overview of the ‘Mirage’ tool but marks the conclusion of the introduction series, and paves the way for more advanced content in the near future.

Incident Response Challenges in Google Cloud

There are three primary challenges when conducting incident response, threat hunting or posture assessments in a Google Cloud ecosystem:

1. Knowledge
2. Access & Collection
3. Analysis

Challenge #1: Knowledge

Lack of knowledge can be a serious obstacle in incident response – as in any cybersecurity field. Simply gaining the relevant access to Google Cloud is a complex process that requires familiarity with the authentication mechanisms, application integration procedures, and the overall Google Cloud environment.

Although Google Cloud has a 9% market share, making it a widely-used tool, there is a scarcity of resources and information that address incident responders’ needs. Such information includes the connection between Google Workspace and GCP, their shared identity management, available and valuable forensics artifacts, and ways to investigate Google Cloud cases.

For these reasons, we created an introductory series of Google blogs – of which this is the third in the series. These blogs are intended to contribute knowledge to the cybersecurity community, and to manage the challenges of working with Google Cloud.

Challenge #2: Access & Collection

• Gaining access to start incident response operations in Google Cloud can be a complex process. This initial effort requires knowledge of targeted evidence sources, the platforms and services where they reside, and the necessary permissions for access. Due to the dynamic nature of incident response, data collection must be efficient, flexible, and scalable, in order to ensure timely response. In addition to collecting data that resides in separate logical locations (i.e., GCP vs Google Workspace), it can be necessary to access user data such as mailbox settings and email messages. An organization that has not purchased services specifically for this purpose has limited options for gaining visibility into user data. To address this challenge, a solution is needed that provides access throughout the Google Cloud ecosystem, supports scalable delegation of authority, and allows for programmatic access.
• Collecting forensic artifacts plays a vital role during the initial phases of incident response. The main challenges associated with collection are flexibility, and scalability. Although the web consoles or Google Cloud CLI (GCloud) are user-friendly, their real-world application is limited. GCloud’s Admin console can be useful for collecting data in Google Workspace or Cloud Identity, but its implementation is limited compared to the use of APIs for data collection. For example, the exporting of log events is restricted to a maximum of 100,000 rows, and must be done separately for each log. When dealing with a large volume of log data, this makes the collection inefficient and may hinder incident response workflows. While using the Admin Console to observe data associated with users, groups, and organizational units is convenient, it is not scalable for more complex inquiries. In addition, the ability to obtain application-specific user settings and data is limited or unavailable in most Google subscription levels. For example, GCloud’s ‘Email Log Search’ option provides only email metadata information and does not include information relating to content or attachments. There are also limitations when extracting GCP data through GCloud commands or the Cloud console. If a company has not already aggregated log data to a centralized location, it is impossible to export historical events from individual projects, folders, and the organization in an efficient and scalable manner. A lack of programmatic access also increases the difficulty of examining basic GCP environment configuration (e.g., IAM role bindings and service configuration), requiring navigation of the Cloud console or scripting GCloud commands.

One of the options that Google offers to centralize the data scattered throughout different locations is through forwarding Google Workspace logs to GCP, using the ‘Share data with Google Cloud services’ option. However, this solution does not fulfill security teams’ needs, as it forwards only specific selected Google Workspace logs and is applicable only to live-stream data, leaving us with the same problem of centralizing all the data in one place.

Challenge #3: Effective Analysis

Public cloud environments are complex and constantly evolving, making it difficult for security teams to handle incidents in Google Cloud. When compared to its market-dominant competitors (e.g., AWS and Azure), there is a lack of publicly-accessible knowledge and resources for incident response and threat hunting in GCloud – especially in terms of how to utilize logs and configurations to identify suspicious or anomalous activity. Effective security operations can be aided by acquiring knowledge of concepts like privilege escalation through impersonation, high-risk API calls, and misconfiguration in ‘crown jewel’ services. Although the concepts of log analysis and architecture review are not new, their implementation in Google Cloud can be different from the implementation in other cloud providers.

Subsequent blogs in this series will delve deeper into the analysis of Google Cloud logs and configurations.

Solutions

In order to handle incident response more effectively in Google Cloud, it is vital to overcome the obstacles relating to Knowledge, Access and Collection, and Analysis. Having identified these challenges, Sygnia team focused the efforts on two subjects:

• Knowledge and Analysis Strategies – develop Sygnia blog posts and webinars for a better understanding of Google Cloud.
• Collection – release Mirage, an open-source collection tool that is flexible and simple to use.

Exploring Sygnia’s Cloud Evidence Collection Tool: Mirage

Sygnia developed the Mirage Python-based tool to assist and improve incident response and threat hunting operations in Google Cloud. Mirage consists of two individual components:

• Assistant: automates access to Google Cloud environments for evidence collection, and removes access once operations have concluded. This script fulfills the required prerequisites for the second component: the Collector script.
• Collector: collects logs, configurations, and user data on-demand from Google Workspace and GCP.

Capabilities Highlights – Why to Use Mirage

The main capabilities of Mirage include the following:

• Aggregate logs and configurations from different Google Cloud components.
• Access user-specific data in Gmail.
• Automate access prerequisites in preparation for evidence collection.
• Obtain significant insights to improve security posture.
• Provide an intuitive and efficient method of collecting specific, or all available logs.

The collection capabilities of Mirage are summarized in the following table:

Additionally, all of the tool’s output is preserved in its original JSON formatting, so that it may be used for further investigation by any analysis platform or tool, such as Splunk, Elastic, or the jq JSON processor.

Methodology – When to Use Mirage

Mirage can be used in various types of projects to collect forensic logs, configurations, and data from Google Cloud. The setup phase takes only a few minutes, and can be initiated either prior to, or at the start of a forensic investigation.

Once a project starts, evidence should be collected based on the project’s type and goal. Following are several guidelines on how to use Mirage:

Access & Collection – Incident Response

• When an incident is detected – whether it is indicated by a Google Alert Center alert, or any source that suggests Google Cloud is involved – logs should be the main priority to focus on.
  • If the incident is identified in Google Workspace, collect the logs relevant to the incident based on the module table available in the Mirage
  • tool’s README file, or collect all the available logs.
  • If the incident involves GCP,collect the logs fromtherelevant resource – for example projects, folders, or organization. If possible, it is recommended to collect logs from specific resources, to minimize the potentially high volume of entries in GCP logs.
• If the investigation lead suggests that emails may be involved, Mirage can be used to collect emails from a suspected mailbox, or to use queries to search for suspicious emails across all mailboxes, if applicable. This capability is important, given that Gmail visibility may be limited by subscription level.

Access & Collection – Threat Hunting

• Define a scope for the threat hunting activity and decide on main threat hunting analytics.
• Based on the defined scope, use Mirage to collect all the available logs and configurations from Google Workspace and/or GCP. If Gmail is in scope, collect mailbox configurations for relevant users, or query Gmail messages using Mirage.

Access & Collection – Posture Assessment

• Risk assessments require inspecting problematic configurationsthat may be exploited by threat actors, internal threats, or mistakes made by over-privileged users. At the start of a risk assessment, define the project’s scope, including what configurations should be inspected.
• Once the scope is decided, use Mirage to collect identity management, Gmail, and GCP configuration data for specific users or for all users.

Analysis – General Investigation

Following any of the above types of project, continue with the following steps after initial evidence collection:

• Once the evidence files are collected by Mirage in JSON format, it is recommended to index them to any SIEM platform before analyzing. For best results, index logs and configurations separately.
• During the investigation, if any additional logs, email data, or configurations from different sources are required, use Mirage again to collect them instantly. It is important to remember that Google Workspace and GCP are linked, and therefore it may be necessary to collect evidence from both platforms. Further demonstration of this concept can be found in the case analyses that will be shared in future blogs.

How Does Mirage Work?

In this section, we cover how Mirage utilizes Google APIs and GCloud commands to gain access to and acquire the relevant evidence from Google Cloud.

Accessing Google Cloud

Mirage utilizes service accounts to access Google Cloud, providing a flexible solution for incident response at scale. Any service account can make authorized API calls by authenticating as itself, or as any user account, through Google’s domain-wide delegation.

The process for gaining access to Google Cloud is detailed in the following diagram:

The Mirage tool creates a separate project and service account in GCP, to enable the clear separation of responsibilities and access controls, and to provide visibility over incident response activities. This approach contributes to making security operations more efficient, effective, and secure.

Collection at Scale

Mirage employs Google API client libraries to programmatically collect logs, configurations, and data in both Google Workspace (or Cloud Identity) and GCP. By interacting with Google APIs, in conjunction with domain-wide delegation of authority, Mirage can bypass any log export restrictions and access data that is not readily available in the web consoles. By making API requests on behalf of a user account, the service account can gather all the relevant evidence, including user-specific data such as mailbox settings, email threads, and email attachments. The use of APIs provides a programmatic way to access data, which can increase the speed and efficiency of acquiring forensic evidence, as well as providing more flexibility and control over the data collected.

When using APIs, we can gather data that is otherwise only made available to the Admin console by purchasing additional services (such as Google Vault), or a higher subscription tier.

Necessary Prerequisites

To use Mirage collection, the following prerequisites are required:

• A service account key file in JSON format, as downloaded after using Mirage Assistant in Google Cloud Shell. Detailed usage explanation can be found in the Mirage Assistant  README file.
• The email address of a Super Admin user,for example: [email protected]. If a Super Admin user cannot be used, choose a dedicated user with the appropriate privileges during collection. The requirements for the dedicated user can be found in Appendix #1: Necessary Permissions for Utilizing Mirage.
• If required, the id of any specific GCP or Workspace resource.

Conclusion and Case Studies – Incident Response with Mirage

The purpose of this blog – along with Sygnia’s previous blogs on the subject and the open-source tool Mirage – is to help organizations overcome incident response challenges in Google Cloud. In this blog, we began by exploring the three categories of challenges that organizations face, and then introduced Mirage, a forensic collection tool for Google Cloud that streamlines evidence collection and provides insights to improve security posture.

In order to gain a better understanding of Mirage functionality, and the value added to organizations that makes use of this tool, it is recommended to combine the reading of this blog with a review of case studies, which will be published in the near future.

The case studies are based on real attacks that were investigated by Sygnia’s IR team, while leveraging Mirage to efficiently collect evidence and explore analysis strategies.

Appendix #1: Necessary Permissions for Utilizing Mirage

If you decide not to use a Super Admin account for the Mirage delegation process, or this account cannot be used, use a dedicated user with the following permissions, based on the desired actions from Mirage:

Contributors: Itay Shohat, Wesley Guerra, Shani Adir

Google Cloud forensics blog series

• Incident Response in Google Cloud: Foundations
• Incident Response in Google Cloud: Forensic Artifacts
• Announcing ‘Mirage’ – New Opensource Tool to Combat Google Cloud Incident Response Challenges
• Proof of Concept: Overcoming Google Cloud Incident Response Issues with ‘Mirage’

This advisory and any information or recommendation contained herein has been prepared for general informational purposes and is not intended to be used as a substitute for professional consultation on facts and circumstances specific to any entity. While we have made attempts to ensure the information contained herein has been obtained from reliable sources and to perform rigorous analysis, this advisory is based on initial rapid study, and needs to be treated accordingly. Sygnia is not responsible for any errors or omissions, or for the results obtained from the use of this Advisory. This Advisory is provided on an as-is basis, and without warranties of any kind.