China-Nexus Threat Group ‘Velvet Ant’ Abuses F5 Load Balancers for Persistence

Key Takeaways

• In late 2023, a large organization was the victim of a serious cyber attack. Sygnia’s forensic investigation into the attack revealed a sophisticated threat actor who exhibited robust capabilities and employed a methodical approach. The evidence gathered suggests the involvement of a China-nexus state-sponsored threat actor.
• Velvet Ant is a sophisticated and innovative threat actor. The investigation confirmed the threat actor maintained a prolonged presence in the organization’s on–premises network for about three years. The overall goal behind this campaign was to maintain access to the target network for espionage.
• The threat actor achieved remarkable persistence by establishing and maintaining multiple footholds within the victim company’s environment. One of the mechanisms utilized for persistence was a legacy F5 BIG-IP appliance, which was exposed to the internet and which the threat actor leveraged as an internal Command and Control (C&C).
• After one foothold was discovered and remediated, the threat actor swiftly pivoted to another, demonstrating agility and adaptability in evading detection.
• The threat actor exploited various entry points across the victim’s network infrastructure, indicating a comprehensive understanding of the target’s environment.
• This incident highlights the importance of establishing resilient defense strategies against sophisticated threats – particularly those posed by state-sponsored groups. A holistic approach to mitigating these threats combines continuous monitoring with proactive response mechanisms – including periodic and systematic threat hunts – alongside stringent traffic controls and system hardening practices for both legacy and public-facing devices. By embracing such an approach, organizations can enhance their ability to detect, deter, and counteract the persistent threat presented by state-sponsored groups.

Introduction

In late 2023, Sygnia responded to a cyber attack targeting a large organization. During the investigation, Sygnia determined that the threat actor exhibited behaviors like those of a China-nexus threat actor. This threat actor, which Sygnia tracks as Velvet Ant, maintained access to the organization’s network for at least three years, and had succeeded in gaining a strong foothold, and intimate knowledge of the network.

Over the course of the attack, the threat actor adeptly used various tools and techniques to infiltrate critical systems and obtain access to sensitive data. The prolonged period of the attack allowed the threat actor to familiarize itself with the complex network infrastructure, enabling it to conceal persistence mechanisms within overlooked areas. Although Sygnia eventually ousted the threat actor from the network, the eradication process resembled a relentless game of cat-and-mouse. Despite Sygnia’s diligent efforts to remediate compromised systems and enhance visibility into hosts and network devices, the threat actor resurfaced time and again through the use of dormant persistence mechanisms in unmonitored systems.

This blog describes the usage of one of those mechanisms, which included leveraging legacy servers and unpatched network appliances.

It all began with PlugX

Throughout the duration of the attack, one of Velvet Ant’s attack techniques was to hijack execution flow, by leveraging different methods such as DLL search order hijacking, Phantom DLL loading and DLL side loading.  

Soon after starting the forensic investigation, Sygnia identified the various tools employed by the threat actor, determined the scope of compromise, and formulated a remediation plan. The plan combined eradication, remediation, and visibility enhancement, which focused on the threat actor’s Tactics, Techniques and Procedures (TTPs).

The plan was executed successfully. Alerts that were configured to identify re-entry attempts were not triggered, and Sygnia’s experts worked with the IT and security teams of the victim company to bolster overall security. However, the success of the remediation and hardening efforts drove the threat actor to focus on legacy operating systems, targeting Windows Server 2003 systems on which the organization’s Endpoint Detection and Response (EDR) product was not installed, and logging was limited. The threat actor resumed activity by utilizing previously deployed malware that remained dormant for months in the victim environment. The malware was identified as PlugX, a well-known remote access Trojan.

PlugX has been used by multiple Chinese state sponsored groups since 2008. The tool is still widely deployed – although its successor, ShadowPad has been in use since 2015. PlugX was primarily designed to provide remote access to infected systems. However, it has a modular plugin system, which provides attackers with a large set of additional capabilities that can be utilized for nefarious purposes.  

The PlugX execution chain in this network consisted of three files: ‘iviewers.exe’, ‘iviewers.dll’ and ‘iviewers.dll.ui’. 

• ‘iviewers.exe’ is a legitimate application called ’OLE/COM Object Viewer‘, that is part of the Windows SDK.
• ‘iviewers.dll’ is the malicious PlugX DLL loader, that is loaded by ‘iviewers.exe’ via DLL search order hijacking.
• ‘iviewers.dll.ui’ contains the actual malicious payload, which is loaded by ‘iviewers.dll’.

When ‘iviewers.exe’ is executed, these three files are copied to a sub-directory with a non-fixed name under ‘C:\ProgramData’ (or ‘C:\Documents and Settings\All Users\Application Data’, on Windows Server 2003 systems), and ‘iviewers.exe’ is installed as a Windows service. Afterwards, several ‘Svchost’ processes are launched, and code is injected into them.

Throughout the investigation, Sygnia acquired memory dumps of those ‘Svchost’ processes, which proved helpful in tracking the threat actor’s activities – especially on legacy servers, where logs were lacking. The memory dumps contained harvested credentials and various commands executed by the threat actor.

Lateral Movement and C&C

When moving laterally to workstations installed with newer Windows versions, the threat actor consistently tampered with the EDR product prior to installing PlugX. On one occasion, the threat actor attempted to disable the EDR product on a target workstation but failed, and then did not proceed to install PlugX. This demonstrates a high level of operational security (OPSEC) awareness.

Impacket, an open-source collection of Python classes, was used for lateral tool transfer, and to execute code remotely on hosts. Specifically, the threat actor employed Impacket’s wmiexec.py, a tool that utilizes the native Windows Management Instrumentation (WMI) to execute remote commands.

The PlugX C&C address, 202.61.136[.]158, was also found in these memory dumps. This allowed the creation of network-based detections, and assisted Sygnia in scoping the incident.

The ‘Svchost’ process launched by ‘iviewers.exe’ binds to a pre-configured local port and accepts TCP connections. It also creates a new rule in the Windows firewall called ‘Network Discovery (SSDP-In)’, to allow these connections. This behavior did not seem to be related to the C&C mechanism, and its purpose was initially unclear.

The purpose of the local firewall rule soon became clear, following a kill-switch operation that initiated a dramatic turn of events.

PlugX is Dead; Long Live PlugX

Equipped with the forensic investigation findings, the Sygnia team again worked with the victim company to eradicate the discovered threats. Dozens of compromised hosts were re-imaged, many of the legacy servers were decommissioned, hundreds of indicators of compromise (IOCs) were blocked across the network, and all the known instances of threat actor access to the network were eradicated.

The victim organization has a large network with complex architecture. The post-breach remediation plan designed by Sygnia included prioritized tasks divided into immediate and mid-term recommendations. As a standard procedure, Sygnia enhanced the organization’s visibility, and established advanced monitoring based on the indicators of attack (IOA).

Several days after eliminating Velvet Ant’s presence, the advanced monitoring capabilities that were deployed in the network proved their value, as alerts were triggered when PlugX was deployed on newly-infected systems. Sygnia’s investigation team quickly analyzed the new infections and noticed a substantial difference in the memory dump acquired from the newly infected hosts – there was no external C&C configured. This was puzzling; PlugX is a remote access tool – how can it be deployed without a C&C server, and still be effective?

Additional forensic analysis revealed that the threat actor reconfigured PlugX to use an internal file server as its C&C and channeled traffic through that server. This defense evasion technique allowed the C&C traffic to blend in with legitimate internal network traffic.

This meant that the threat actor deployed two versions of PlugX within the network. The first version, configured with an external C&C server, was installed on endpoints with direct internet access, facilitating the exfiltration of sensitive information. The second version did not have a C&C and was deployed exclusively on legacy servers.

This left us with unanswered questions. How did the threat actor control the legacy file server? And how did they manage to restore access to the organization despite the comprehensive remediation activities? To answer these questions, Sygnia’s incident response team set up additional network and host-based monitoring tools on the file server and closely monitored for any suspicious activities.

As shown in figure 7, PlugX created a local firewall rule on compromised endpoints and listened on a high, random port number. On the file server, port 13742 was opened. Analyzing network connections to the file server identified an internal IP address communicating with it on the same port. The internal IP address belonged to an F5 load balancer.

This load balancer was not part of previous remediation efforts, because it was not supposed to be operational in the production network. One team in the organization started deploying the F5 solution in the network a long time ago as part of a disaster recovery plan (DRP), but the project was never completed.

F5 BIG-IP – The Perfect Place to Hide

F5 BIG-IP appliances occupy a trusted position within the network architecture, often placed at the perimeter or between different network segments. By compromising such a device, attackers can exert significant control over network traffic without arousing suspicion. Moreover, while organizations frequently gather logs from network devices, their attention is predominantly directed towards application-level logs such as traffic records and alerts. Unfortunately, operating system logs are often overlooked, either due to vendor constraints or a lack of recognition regarding their significance. As a result, a backdoor hidden within the F5 appliance can evade detection from traditional log monitoring solutions.

The compromised organization had two F5 BIG-IP appliances which provided services such as firewall, WAF, load balancing and local traffic management. These appliances were directly exposed to the internet, and both of which were compromised.

Both F5 appliances were running an outdated, vulnerable, operating system. The threat actor may have leveraged one of the vulnerabilities to gain remote access to the appliances. However, visibility limitations hinder the ability to identify exactly how the appliances were compromised.​

Forensic investigation of the appliances revealed a reverse SSH tunnel connection to the same C&C IP address that was previously blocked on the corporate firewalls during the eradication efforts. Since the appliances were not located behind the main corporate firewall, the traffic was not blocked.

Analyzing the active network connections on the F5, an established connection was observed between the appliance and the file server, on the port PlugX was listening on. This complemented the analysis made on the file server itself (see figure 8).

Pivoting From the F5 Appliance

The figure below depicts the stages of the threat actor’s working flow.

(1) The malware running on the device polls the C&C server once an hour. After connecting to the C&C server (2), the threat actor sends commands to the compromised appliance, including for the creation of a reverse SSH tunnel (3). Once the tunnel with the appliance is established, the threat actor connects to the file server that is infected with PlugX (4).

The PlugX instance on the compromised file server was used by the threat actor as an internal C&C server. From this server, the threat actor conducted reconnaissance activities, deployed additional instances of the PlugX onto legacy servers by leveraging Impacket’s WmiExec (5).

Recording the traffic between the F5 appliance and the file server revealed additional details about the threat actor’s TTPs, as the traffic was sent over SMB, and was not encrypted. The following figures show the interactive activities conducted by the threat actor prior to deploying additional PlugX instances.

The threat actor’s first step was to enumerate the active network connections on the targeted server:

After selecting a new server for PlugX deployment, the threat actor listed files in different folders, presumably in order to choose where to deploy the malware.

The threat actor then chose a folder, and leveraged WmiExec to transfer PlugX over SMB from the file server to the target server.

Additional Malware on the F5 Appliances

Forensic analysis of the F5 appliances identified four binaries deployed by the threat actor:

1. VELVETSTING – a tool that connects to the threat actor’s C&C once an hour, searching commands to execute. The threat actor used the IP address 202.61.136[.]158:8443 as a C&C and the commands were encoded with the passphrase ‘1qaz@WSXedc’. Once the tool received a command, it was executed via ‘csh’ (Unix C shell).

2. VELVETTAP – a tool with the ability to capture network packets. The binary was executed on the F5 appliance with the argument ‘mgmt’, which is the name of the internal NIC of the F5 device.

To maintain persistence on the F5 appliance, the threat actor added both VELVETSTING and VELVETTAP to the ‘/etc/rc.local’ file. ‘rc.local’ is executed as part of the system’s startup process, which means that any commands or scripts listed in it will be executed automatically whenever the system boots up.

3. SAMRID – identified as ‘EarthWorm’, an open-source SOCKS proxy tunneller available on GitHub. The tool was utilized in the past by different Chinese state-sponsored groups, such as ‘Volt Typhoon’, ‘APT27’ and ‘Gelsemium’. The tool was not running on the device at the time of investigation.
4. ESRDE – a tool with similar capabilities to that of ‘VELVETSTING’, but with minor differences, such as using bash instead of ‘csh’. The tool was not running on the device at the time of investigation.

A Note on Attribution

The attack described in this blog demonstrates a single attack flow of Velvet Ant, a threat actor that exhibited characteristics like those of a China-nexus state sponsored threat actor. These include but are not limited to target selection (industry and geo-location), clear definition of goals, targeting of network devices, exploitation of vulnerabilities, and a toolset that includes the ShadowPad and PlugX malware families, and utilization of DLL side-loading techniques. The threat actor was extremely persistent and remained active in the compromised network for about three years, despite several eradication attempts throughout that period.

The modus operandi of Chinese-nexus Intrusion Sets includes shared tools, infrastructure, and sometimes manpower – for example via shared contractors. This mode of operation makes it difficult for Sygnia to attribute the attack to any previously publicly-reported group. Moreover, the visibility available in the network prevents Sygnia from ruling out a ‘false-flag’ operation conducted by a different Advanced Persistent Threat (APT) group.

Defending Against Velvet Ant

The defense strategies presented below are designed to provide organizations with robust measures for countering threats related to C&C and the malicious weaponization of legacy and edge devices strengthening organizational cyber security defenses and strengthening resilience against advanced persistent attacks.

• Limit outbound internet traffic: Outbound internet connections should be restricted across servers, workstations, and edge devices, to prevent attackers from maintaining access via Command and Control (C&C) channels – similar to the tactic of utilizing the PlugX loader. Configure perimeter firewalls to allow only the necessary outbound connections, placing internet-facing devices such as load balancers behind these firewalls to prevent direct public access.
• Limit lateral movement throughout the network: Performing lateral movement within networks is a technique frequently employed by threat actors using tools like WmiExec. To mitigate this risk, it is essential to rigorously control traffic over common management ports. These include SMB on port 445, RPC on port 135, WinRM on ports 5985-5986, RDP on port 3389, and SSH on port 22. Access should be strictly limited to hosts with explicit authorization. Although implementing such controls poses challenges in complex network environments, the security benefits are substantial. Mitigation strategies may include deploying host-based firewalls,establishing effective network segmentation, and adopting cutting-edge microsegmentation technologies.
• Enhance security hardening of legacy servers: Prioritize decommissioning and replacement of legacy systems. Tighten control over legacy servers, which are prime targets for cyberattacks due to their weaker defenses. Enhance security by installing Endpoint protection tools that support older systems such as Windows Server 2003, or using alternatives like Sysmon for comprehensive monitoring. Strengthen protections by limiting access to essential staff, enforcing strict network segmentation, and monitoring traffic to and from these servers. These steps considerably reduce the vulnerabilities inherent in older server infrastructures.
• Mitigate credential harvesting: Implement Endpoint Detection and Response (EDR) systems for continuous monitoring and interception of malicious actions. Ensure that EDR sensors are configured with anti-tampering features, and are consistently operational and current, as attackers might attempt to neutralize or circumvent the sensors. Apply Protected Process Light (PPL) to LSASS, and activate Windows Credential Guard for credential security.
• Protect public-facing devices: To improve the security of public-facing edge devices such as F5 BIG-IP load balancers, it is essential to implement a proactive security strategy that includes external attack surface and asset management, patch management, Intrusion Detection and Prevention Systems, and robust security and integrity monitoring. Additionally, migration to cloud-based solutions such as SASE or cloud-native load balancers can reduce the risk of running vulnerable, exploitable edge devices. For comprehensive guidance on establishing such defenses, refer to Sygnia’s blog “Defending Your Network Edge Against the Next Zero-Day Exploit”, which outlines key measures for enhancing the resilience of network edges against threats demonstrated by this threat actor, and other emerging threats.

Appendix I: Indicators of Compromise

Appendix II: MITRE ATT&CK Matrix Mapping

1. Initial Access
   1. T1133 – External Remote Services
2. Execution
   1. T1047- Windows Management Instrumentation
   2. T1059.008 – Command and Scripting Interpreter: Network Device CLI
   3. T1569.002 – System Services: Service Execution
3. Persistence
   1. T1037.004 – Boot or Logon Initialization Scripts: RC Scripts
   2. T1133 – External Remote Services
   3. T1078.002 – Valid Accounts: Domain Accounts
   4. T1078.003 – Valid Accounts: Local Accounts
4. Privilege Escalation
   1. T1078.002 – Valid Accounts: Domain Accounts
5. Defense Evasion
   1. T1574.001 – Hijack Execution Flow: DLL Search Order Hijacking
   2. T1562.004 – Impair Defenses: Disable or Modify System Firewall
   3. T1055 – Process Injection
   4. T1070.006 – Indicator Removal: Timestomp
   5. Masquerading: Match Legitimate Name or Location (T1036.005)
6. Credential Access
   1. T1003.001 – OS Credential Dumping: LSASS Memory
7. Discovery
   1. T1087.002 – Account Discovery: Domain Account
   2. T1083 – File and Directory Discovery
   3. T1135 – Network Share Discovery
   4. T1018 – Remote System Discovery
   5. T1082 – System Information Discovery
   6. T1016 – System Network Configuration Discovery
   7. T1040 – Network Sniffing
8. Lateral Movement
   1. T1021.002 – Remote Services: SMB/Windows Admin Shares
   2. T1021.004 – Remote Services: SSH
   3. T1570 – Lateral Tool Transfer
9. Collection
   1. T1039 – Data from Network Shared Drive
10. Command and Control
    1. T1572 – Protocol Tunneling
    2. T1090.001 – Proxy: Internal Proxy
    3. T1132.001 – Data Encoding: Standard Encoding
    4. T1071.001 – Application Layer Protocol: Web Protocols
11. Exfiltration
    1. T1048 – Exfiltration Over Alternative Protocol

If you were impacted by this attack or are seeking guidance on how to prevent similar attacks, please contact us at [email protected] or our 24-hour hotline +1-877-686-8680.

This advisory and any information or recommendation contained here has been prepared for general informational purposes and is not intended to be used as a substitute for professional consultation on facts and circumstances specific to any entity. While we have made attempts to ensure the information contained herein has been obtained from reliable sources and to perform rigorous analysis, this advisory is based on initial rapid study, and needs to be treated accordingly. Sygnia is not responsible for any errors or omissions, or for the results obtained from the use of this Advisory. This advisory is provided on an as-is basis, and without warranties of any kind.