XZ Vulnerability: Understanding Complex Supply Chain Attacks

We covered the xz vulnerability in a previous article, but it's interesting to say a little more about the modality of this attack.

xz is a popular tool for compressing data. It's quite good at it, it is reasonably fast (not the fastest) but it can achieve a very high compression ratio. For this reason, is widely adopted by many organizations.

This tool, which uses the LZMA algorithm, was created as a pet project by Lasse Collin. It was probably never intended to be a big deal, just a toy project to have some fun.

Nevertheless, its wide adoption made it a target.

Timeline: How it Happened

2021

In 2021 a GitHub account (JiaT75) was created and started to commit to libarchive. Those commits are now regarded as highly suspicious, as they add some potential vulnerabilities, but were merged without any discussion.

2022

In 2022, an innocent patch was submitted via a mailing list to xz. Another person, Jigar Kumar - probably fake, pressed Lasse for merging the patch and for adding another maintainer, since he (Lasse) couldn't stand the pressure.

Progress will not happen until there is new maintainer. XZ for C has sparse commit log too. Dennis you are better off waiting until new maintainer happens or fork yourself. Submitting patches here has no purpose these days. The current maintainer lost interest or doesn't care to maintain anymore. It is sad to see for a repo like this.

And again:

With your current rate, I very doubt to see 5.4.0 release this year. The only progress since april has been small changes to test code. You ignore the many patches bit rotting away on this mailing list. Right now you choke your repo. Why wait until 5.4.0 to change maintainer? Why delay what your repo needs?

2023

In January 2023 Lasse gave up and added JiaT75 to the maintainers. Two months later, JiaT75 disabled one security tool (oss-fuzz). This appears to be deliberate to mask future changes.

2024

On the 15th of February, a commit to ignore the file build-to-host.m4 (via .gitignore) is added; one week later (on the 23rd) a couple of different files, hosting the actual obfuscated payload, were added as part of the "test" pipeline.

On the 24th of February, version 5.6.0 is released. While missing on the git archive, the actual tarball has the malicious build-to-host.m4 file that, during the build process, injects the code present in the other two "test" files into the library.

If you build the xz library from the tarball, the malicious code is triggered and injected into the final library artifact. Note that this only happens if the system is building a RPM or a DEB archive, so it is clearly targeted at major Linux distributions.

The proof is that a few days later, a call to include the new release is made on Debian, Ubuntu, and Fedora.

How it Works

From what is so far known, the injected code just triggers only when the library is loaded from SSH login process.

Technically, it hooks the RSA_public_decrypt function, extract a parameter from the RSA key that the client presented, it decodes and checks it using hardcoded keys, and then directly executes the payload.

Note that "plain" OpenSSH is not using xz library, so it shouldn't be affected. Unfortunately, most distributions use a patched version of OpenSSH, to integrate it with systemd. And systemd does use xz libs, so the resulting process is vulnerable.

How it was Discovered

Andres Freund spotted that the CPU usage was oddly high during login, investigated the issue, and found out that the liblzma (part of xz package) was compromised. He quickly realized that the backdoor was not in the Debian package he was using, but that this was upstream.

Note that the impact on his system was just 500ms per login attempt so... it was not an easy spot.

Impact of Attack

Since the backdoor was detected quickly, no major distribution has the vulnerable package in their stable release. Only rolling distribution or nightly/beta versions are likely to be affected.

Those include RedHat Fedora Rawhide, RedHat Fedora 40 beta, openSUSE Tumbleweed, openSUSE MicroOS, and Kali Linux. Archlinux shipped a vulnerable version, but the payload was likely not included since it is not rpm nor deb based.

Conclusion

We dodged a bullet here. This attack was well orchestrated with years of planning, targeting an essential component handled by a single person in their free time.

It is a clear textbook example of how to perform a supply chain attack, mixing social engineering, obfuscation techniques, advanced encryption, and patience.

Everything here, from the planning to the ciphers used, suggests a nation-state attack. The eagle eye of a single developer has stopped what could be a real disaster.

For those thinking that open source is a risk, note that if the component were to be closed, it would be impossible to detect the threat until it was too late.

Not for the first time, we are reminded that our software infrastructure is fragile and that attacks on the supply chain can be devastating.

It's becoming more and more important to be well aware of the components that are used by our software infrastructure, to be able to react to threats like this.