Some no-name came and without any problems asked to become a maintainer in a project used in almost any distro, took it over, put a backdoor in there and no one had any questions? In this case, everything turned out thanks to pure chance. Noname screwed up his backdoor, which attracted the attention of a guy from Microsoft, and out of boredom, he dug up what was what. And if I hadn’t messed up, or that guy from Microsoft decided to go drink beer instead of poking around in the xz code, then no one would have discovered anything. It’s scary to imagine how many of these nonames are sitting in all these thousands of open source projects, waiting in the wings to roll out a malicious patch.
Apparently the backdoor reverts back to regular operation if the payload is malformed or the signature from the attacker’s key doesn’t verify. Unfortunately, this means that unless a bug is found, we can’t write a reliable/reusable over-the-network scanner.
Maybe not. But it does mean that you can write a crawler that slams the door shut for the attacker on any vulnerable systems.
EDIT: Oh, maybe he just means that it reverts for that single invocation.
I know this is an issue fraught with potential legal and political BS, and it’s impossible to check everything without automation these days, but is there an organization that trains and pays people to work as security researchers or QA for open source projects?
Basically, a watchdog group that finds exploitable security vulnerabilities, and works with individuals or vendors to patch them? Maybe make it a publicly owned and operated group with mandatory reporting of some kind. An international project funded by multiple governments, where it’s harder for a single point of influence to hide exploits, abuse secrets, or interfere with the researchers? They don’t own or control any code, just find security issues and advise.
I don’t know.
Just thinking that modern security is getting pretty complicated, with so many moving parts and all.
Since the actual operation of the liblzma SSH backdoor payload is still unknown, there’s a protocol for securing your impacted systems:
• Consider all data, including key material and secrets on the impacted system as compromised. Expand the impact to other systems, as needed (for example: if a local SSH key is used to access a remote system then the remote system must be considered impacted as well, within the scope the key provides).
• Wipe the impacted host and reinstall it from scratch. Use known good install that does not contain the malicious payload. Generate new keys and passwords. Do not reuse any from the impacted systems.
• Restore configuration and data from backups, but from before the time the malicious liblzma package was installed. However, be careful not to allow potentially leaked credentials or keys to have access to the newly installed system (for example via $HOME/.ssh/authorized_keys).
This handles the systems themselves. Unfortunately any passwords and other credentials stored, accessed or processed with the impacted systems must be considered compromised as well. Change passwords on web sites and other services as needed. Consider the fact that the attacker may have accessed the services and added ways to restore access via for example email address or phone number in their control. Check all information stored on the services for correctness.
This is a lot of work, certainly much more than just upgrading the liblzma package. This is the price you have to pay to stay safe. Just upgrading your liblzma package and hoping for the best is always an option, too. It’s up to you to decide if this is a risk worth taking.
This recovery protocol might change somewhat once the actual operation of the payload is figured out. There might be situations where the impact could be more limited.
As an example: If it turns out that the payload is fully contained and only allows unauthorized remote access via the tampered sshd, and the host is not directly accessible from the internet (the SSH port is not open to internet) this would mean that it might be possible to clean up the system locally without full reinstall.
However, do note that the information stored on the system might have still been leaked to outside world. For example leaked ssh keys without a passphrase could still afford the attacker access to remote systems.
This is a long con, and honestly the only people at fault are the bad actors themselves. Assuming Jia Tan’s GitHub identity and pgp key weren’t compromised by someone else, this backdoor appears to be the culmination of three years of work.
could a Flatpak contain one of the backdoored builds of xz or liblzma? Is there a way to check? Would such a thing be exploitable, or does this backdoor only affect ssh servers?
The base runtime pretty much every Flatpak uses includes xz/liblzma, but none of the affected versions are included. You can poke around in a base runtime shell with flatpak run --command=sh org.freedesktop.Platform//23.08 or similar, and check your installed runtimes with flatpak list --runtime.
23.08 is the current latest version used by most apps on Flathub and includes xz 5.4.6. 22.08 is an older version you might also still have installed and includes xz 5.2.12. They're both pre-backdoor.
It seems there's an issue open on the freedesktop-sdk repo to revert xz to an even earlier version predating the backdoorer's significant involvement in xz, which some other distros are also doing out of an abundance of caution.
So, as far as we know: nothing uses the backdoored version, even if it did use that version it wouldn't be compiled in (since org.freedesktop.Platform isn't built using Deb or RPM packaging and that's one of the conditions), even if it was compiled in it would to our current knowledge only affect sshd, the runtime doesn't include an sshd at all, and they're still being extra cautious anyway.
One caveat: There is an unstable version of the runtime that does have the backdoored version, but that's not used anywhere (I don't believe it's allowed on Flathub since it entirely defeats the point of it).
I will laugh out loud if the “fixed” binary contains a second backdoor, but one of better quality. It’s reminiscent of a poorly hidden small joint, which is naturally found, and then bargaining, apologizing and making amends begin. Although now it is generally not clear where the code is more proven.
If you’re using xz version 5.6.0 or 5.6.1, please upgrade asap, especially if you’re using a rolling-release distro like Arch or its derivatives. Arch has rolled out the patched version a few hours ago.
Gentoo just reverted back to the last tar signed by another author than the one seeming responsible for the backdoor. The person has been on the project for years, so one should keep up to date and possibly revert even further back than just from 5.6.*. Gentoo just reverted to 5.4.2.
I think that was a precaution. The malicious build script ran during the build, but the backdoor itself was most likely not included in the resuling package as it checked for specific packaging systems.
rolling release (because it was caught so quickly that it hasn’t made its way into any cadence based distro yet)
using the upstream Makefile task to build a RPM or DEB (because the compromised build script directly checks for that and therefore doesn’t trigger for a destdir build like Gentoo’s or Arch’s)
using the upstream provided tarball as opposed to the one GitHub provides, or a git clone (because only that contains the compromised Makefile, running autotools yourself is safe)
Points 1 and 2 mean that only rolling release RPM and DEB distros like Debian Sid and Fedora are candidates. I didn’t check if they use the Makefile and the compromised tarballs.
Also, even aside from the attack code here having unknown implications, the attacker made extensive commits to liblzma over quite a period of time, and added a lot of binary test files to the xz repo that were similar to the one that hid the exploit code here. He also was signing releases for some time prior to this, and could have released a signed tarball that differed from the git repository, as he did here. The 0.6.0 and 0.6.1 releases were contained to this backdoor aimed at sshd, but it’s not impossible that he could have added vulnerabilities prior to this. Xz is used during the Debian packaging process, so code he could change is active during some kind of sensitive points on a lot of systems.
It is entirely possible that this is the first vulnerability that the attacker added, and that all the prior work was to build trust. But…it’s not impossible that there were prior attacks.
The malicious code attempts to hook in to libcrypto, so potentially other services that use libcrypto could be affected too. I don’t think extensive research has been done on this yet.
SSH doesn’t even use liblzma. It’s pulling in the malicious code via libsystemd, which does use liblzma.
Edit: “crypto” meaning cryptography of course, not cryptocurrency.
Add comment