On March 29, right before Easter weekend, we received notifications about something unusual happening with the open-source project XZ Utils, which provides lossless data compression on virtually all Unix-like operating systems, including Linux.
The initial warning was sent to the Open Source Security mailing list sent by Andres Freund, who discovered that XZ Utils versions 5.6.0 and 5.6.1 are impacted by a backdoor. A few hours later, the US government’s CISA and OpenSSF warned about a critical problem: an installed XZ backdoored version could lead to unauthorized remote access.
This came as quite a shock because it perfectly described a software supply chain doomsday scenario. XZ Utils has heavy repercussions on OpenSSH security, which creates more urgency in detecting and removing backdoor versions (CWE-506: Embedded Malicious Code).
XZ Utils is heavily used in embedded systems and firmware development across many different ecosystems. Luckily, in this particular case, the slow deployment of the device updates played a positive role, and so far we haven’t seen any impacted devices. The biggest impact and the major target was the general Linux ecosystem, which powers the modern cloud infrastructure.
According to Repology, many Linux distributions are impacted by CVE-2024-3094. But most of them, with quick industry-wide community-driven coordination, have been rolled back to a known safe version of XZ Utils. The main question is how quickly deployed backdoored versions will be deactivated and detected in the field. Most of the detection tools we have seen have focused on simple version checks, hash-based detection of the backdoored component, or YARA rules of the hardcoded unique string constants. All of these could lead to massive alert fatigue and false positives at scale by overloading security teams.
The figure below shows the original XZ Utils backdoored component distributed in the Open Source Security mailing list by Andres. We can see the detection statistics from VirusTotal which shows quite a few.
This figure shows almost zero detection from VirusTotal for recompiled implanted versions from one of the common Linux distributions. This demonstrates outdated methods such as byte string matching and blacklisting known bad file hashes are of limited use when checking for particular code properties.
The Binarly Research team decided to dive deeper into the details and prioritize the proper approach for the generic detection of such a backdoor implantation to help find the backdoored binaries with confidence.
Software supply chain incidents are complicated, specifically when they impact the entire ecosystem. Finding the needle in a haystack is hard and it’s near impossible to do it manually. It requires a different type of mindset and approach built-in by design to prevent software supply chain issues. Existing source code analysis tools do not look into a compiled software package that ships to the customers or upstream to major Linux ecosystems like in this XZ case.
This is clearly a very complex state-sponsored operation with impressive sophistication and multi-year planning. Such a complex and professionally designed comprehensive implantation framework is not developed for a one-shot operation. It could already be deployed elsewhere or partially reused in other operations. That’s exactly why we started focusing on more generic detection for this complex backdoor.
One of the core techniques used by the XZ backdoor to gain initial control during execution is the GNU Indirect Function (ifunc) attribute for the GCC compiler to resolve indirect function calls in runtime. The implanted backdoor code initially intercepts or hooks execution. It modifies ifunc calls to replace a check "is_arch_extension_supported" which should simply invoke "cpuid" to insert a call to "_get_cpuid" which is exported by the payload object file (i.e., liblzma_la-crc64-fast.o) and which calls malformed _get_cpuid() which is implanted into the code shown in the figure below.
To generically detect such implantation, we decided to focus on ifunc transition behavioral analysis with our Binary Intelligence technology. The detection method is completely based on static analysis, where we are detecting generically the tampering of control flow graph transitions, the transition we mark as suspicious is depicted via IDA's debugger below.
Such detection methods could expose potential control flow tampering during the implantation of malicious ifunc resolvers. This technique works generically and will be able to detect invariants or reuse of the payloads in other software supply chain attacks. In the figure below you can see Binary Intelligence technology in action.
To help the industry mitigate this threat, Binarly has created the free scanning tool available to the entire security community to help expose potential other incidents or find the modifications of the malicious payloads. The detection method we use has close to zero false positive rate.
The XZ backdoor scanner is available for free at XZ.fail.
For more in-depth detection and remediation guidance, the Binarly Transparency Platform has the XZ detection deployed making it simple to find this malicious activity at scale and take appropriate action to keep your software supply chain protected.
The Binarly team is constantly working to protect the software supply chain and reduce the attack surfaces of our customers industry-wide by delivering innovative technologies to the market.
Are you interested in learning more about The Binarly Transparency Platform or other solutions? Don't hesitate to contact us.