Linux was hit with a serious supply chain attack involving XZ Utils, a set of tools and libraries most Linux distros use for data compression.

Andres Freund, a Microsoft engineer, discovered the issue after noticing “logins via ssh became a lot slower,” as well as “ssh taking a lot of CPU,” and “valgrind errors.” Upon further investigation, Freund discovered that malicious code had been injected into the XZ utilities—not Debian’s package, but the upstream package used by many distros.

According to Freund, the exploit appears to be designed to give unauthorized remote access via SSH:

I have not yet analyzed precisely what is being checked for in the injected code, to allow unauthorized access. Since this is running in a pre-authentication context, it seems likely to allow some form of access or other form of remote code execution.

How This Happened

Supply chain attacks are the most devastating type of cyberattack because compromising a single application, utility, or library can impact dozens, hundreds, or even thousands of downstream projects. Supply chain attacks are also among the most difficult to pull off given the lengths companies and organizations go to secure their software. As a result, many supply chain attacks depend on brute force attacks or social engineering.

In contrast, XZ Utils was compromised by the very person who had authority to make and commit changes to the project, Jia Tan. Lasse Collin is the primary XZ maintainer, but had had become increasingly limited “due to long term mental health issues,” as well as “some other things.”

Meanwhile, Jia Tan spent years gaining Collin’s trust, assisting him behind the scenes, and contributing to the project in meaningful and legitimate ways. As a result, when Collin needed to step away, he appointed Tan a co-maintainer, essentially giving him control of the project.

Once he had control, Tan began a slow, patient, coordinated effort to plant a backdoor in the software, doing it in stages and taking steps to circumvent detection. The malicious code was also designed to only target Debian and RPM-based distros. In fact, the backdoor was so expertly inserted into the code that it was only discovered because it degraded SSH performance so much.

To matters even worse, Tan actively engaged with Ubuntu and Fedora maintainers in an effort to get the backdoored version of XZ included in upcoming versions of those distros.

What Version of XZ Was Compromised, and What Distros Are Impacted

Research is still ongoing, but it appears that version 5.6.0 and 5.6.1 of XZ are compromised, while the 5.4.x series appears to be safe. As a result, there are relatively few production distros that are impacted.

openSUSE Tumbleweed and MicroOS

openSUSE’s rolling release distros, both Tumbleweed and MicroOS were both impacted, although the project has already taken mitigation steps:

openSUSE Maintainers have rolled back the version of xz on Tumbleweed on March 28th and have released a new Tumbleweed snapshot (20240328 or later) that was built from a safe backup.

The reversed version is versioned 5.6.1.revertto5.4 and can be queried with rpm -q liblzma5.

Fedora

Red Hat is warning that the development Fedora Rawhide instances included the compromised versions. While the company does not believe the beta builds of Fedora 40 were compromised, users are still urged to downgrade the version of XZ:

PLEASE IMMEDIATELY STOP USAGE OF ANY FEDORA RAWHIDE INSTANCES for work or personal activity. Fedora Rawhide will be reverted to xz-5.4.x shortly, and once that is done, Fedora Rawhide instances can safely be redeployed. Note that Fedora Rawhide is the development distribution of Fedora Linux, and serves as the basis for future Fedora Linux builds (in this case, the yet-to-be-released Fedora Linux 41).

At this time the Fedora Linux 40 builds have not been shown to be compromised. We believe the malicious code injection did not take effect in these builds. However, Fedora Linux 40 users should still downgrade to a 5.4 build to be safe. An update that reverts xz to 5.4.x has recently been published and is becoming available to Fedora Linux 40 users through the normal update system. Concerned users can force the update by following the instructions at https://bodhi.fedoraproject.org/updates/FEDORA-2024-d02c7bb266.

Debian

Debian maintainers announced that Debian Testing and Unstable (Sid) were both impacted, but no version of Debian Stable was impacted:

Right now no Debian stable versions are known to be affected. Compromised packages were part of the Debian testing, unstable and experimental distributions, with versions ranging from 5.5.1alpha-0.1 (uploaded on 2024-02-01), up to and including 5.6.1-1. The package has been reverted to use the upstream 5.4.5 code, which we have versioned 5.6.1+really5.4.5-1.

Users running Debian testing and unstable are urged to update the xz-utils packages.

Ubuntu

Canonical says no production version of Ubuntu was impacted. Development builds of the upcoming 24.04 release did include the backdoored XZ libraries, but they have been removed, ensuring 24.04 will not be impacted either.

The affected version of xz-utils was only in noble-proposed, and was removed before migrating to noble itself. No released versions of Ubuntu were affected by this issue.

Linux Mint

Because Linux Mint is based on Ubuntu 22.04—and the Debian version is based on the current Debian 12—no version of Mint is impacted by the backdoor.

Arch

Some Arch systems were impacted, with the maintainers urging users to upgrade immediately.

Kali Linux

Kali maintainers say the distro was affected between March 26 and March 29, when the backdoored version of XZ was available.

How to Know For Sure If Your Installation Is Compromised

The easiest way to know if your installation is compromised is to run the following command in a terminal. Bear in mind, in many cases it is not safe to call a compromised package directly, but the unique nature of this particular one makes it safe to do so.

xz –version

In my case, running Linux Mint Debian Edition (LMDE), I see the following results:

xz (XZ Utils) 5.4.1
liblzma 5.4.1

This shows that LMDE is using a version of XZ that is believed to be safe, at least as of the time of writing.

What Happens Next

Researchers are continuing to analyze the situation and determine determine the extent of the impact. Although version 5.4.x is believed safe, at least some Debian maintainers are calling for a rollback to 5.3.1, since that appears to be the last version that was released primarily by Collin. Debian has also launched a multi-team task force to determine exactly how many of its Testing and Unstable users were impacted.

The bigger question is who is ultimately responsible and what the Linux community should do moving forward.

The general consensus is that Jia Tan is likely working as a state-sponsored agent. The level of skill, patience, and maneuvering that went into implementing the backdoor seem to go beyond a single hacker. What’s more, the fact that the malicious code was designed to target only Debian and RPM-based distros is telling, since those distros are the ones most commonly used in business, enterprise, and government applications.

The Linux community clearly must come up with a solution to address the issue this incident has raised. While care is taken not to accept code from unknown, untested contributors, Tan’s years of work embedding himself as a legitimate maintainer is the worst-case-scenario, raising questions about what measures can/should be taken to ensure maintainers can be trusted.

The incident also raises questions about how source code should be included. According to Freund, “one portion of the backdoor is solely in the distributed tarballs,” as opposed to the source code on GitHub. Some prominent Linux users, such as Brodie Robertson in the video below, are calling for distros to stop using tarballs to include packages and only use direct source code, a measure that would make it easier to catch this kind of issue.

While the investigations will likely continue for some weeks, there is no denying that this incident is one of the worst issues that will impact Linux at some point. Although the practical impact is, thankfully, limited in scope, the potential impact if the exploit had not been caught is difficult to comprehend. If the backdoor had slipped into the official releases of Ubuntu 24.04, Fedora 40, and remained undetected in openSUSE, Arch, Kali, and others, there is almost no limit to the damage that would be caused.

As stated, just as important are the questions raised about how Linux maintainers should proceed from here and what measures should be taken to keep this type of situation from ever being repeated.

While this incident can largely be chalked up to a near-miss, there’s no denying its effects will be felt for years to come.