HT to @wdormann here - somebody has backdoored the open source project XZ which has downstream impacts.
For example, although OpenSSH doesn’t use XZ, Debian patch OpenSSH and introduced a dependency which translates as the XZ changes introducing a sshd authentication bypass backdoor it appears.
One dude bothered to investigate in his free time about why ssh was running slow, so it was caught fairly early - i.e. hopefully before distros started bundling it.
Worryingly it looks like the backdoor comes via one of the two main devs and dates back over a month from their GitHub account, with legit commits too - XZ is used in systemd so this one might play out for a while.
I suspect distros probably want to roll XZ back to around January 2024, stop bundling updates until the developer is removed in GitHub or a logical explanation can be given, and somebody needs to fund a code review of it.
Back in 2022 a host of characters appeared and basically bullied the creator of the XZ project to hand it over to somebody else - at the time the guy cited mental health issues around not updating the project quickly.
At the time he was already talking about maybe handing over to the account who years later introduced the backdoor.
In mid 2023 said account introduced a change to Google’s OSS Fuzzer to weaken detection for XZ.
Somebody played a years long game of Jenga and lost.
Before everybody high fives each other, this is how the backdoor was found: somebody happened to look at why CPU usage had increased in sshd, and did all the research and notification work themselves. By this point the backdoor had been there for a month unnoticed.
I’ve made the joke before that if GCHQ aren’t introducing backdoors and vulns in open source that I want a tax refund. It wasn’t a joke. And it won’t be just be GCHQ.
@GossiTheDog
One could argue that your tax money should also be spent by GCHQ to happen to look into increased CPU usage after some weird lib update in all the places where they didn't plant anything.
Really good timeline of what is known to have happened so far. It looks like the rogue developer deliberately introduced a vulnerability in other package, too - I haven’t seen anybody else mention this.
Reading the dev’s GitHub history, they’ve been making changes to other open source projects too around compression. It also appears they/somebody involved has other accounts, too.
How far the rabbit hole goes - back in 2021 they deliberately introduced a risky change in the compression library libarchive. Nobody noticed. This is shipped in a ton of systems: https://github.com/libarchive/libarchive/pull/1609
Whoever the threat actor is knows what they are doing as they’ve gone after chained dependencies around compression.
@GossiTheDog The very recent zstd fork branch updates agree with the assessment that the compression ecosystem as a whole was the domain this threat actor was playing in:
Also since there’s a lot going on here, up thread I mentioned a 2015 minor bug in Google’s OSS Fuzzer (security testing tool) - the threat actor deliberately introduced the bugged function into XZ, then used that to get an exception in OSS Fuzzer’s code to stop scanning of XZ.
I’ve just been looking at the actual backdoor for a few hours with greater minds than me, it’s incredibly complex - it basically piggy backs RSA key RCE inside sshd as a Trojan horse. Somebody/bodies spent $$ on this.
@ohmu@GossiTheDog doubt any other entity has 2+ years of timeframe to slowly psyops a poor maintainer into handing over control and insert the most subtle of vulnerabilities into the test/build chain that require world-class vuln researchers to understand
@analogist@ohmu@GossiTheDog + all the work to identify this library and it’s maintainer as vulnerable to “attack”. They had to work out that systemd loaded extra libraries into sshd, then examine all of them to identify
-small projects trusted as stable
-single mostly inactive maintainer.
They could code & test the exploit in parallel with the takeover -after doing some PoC to show that any exploit was possible
@analogist@ohmu@GossiTheDog if I was in a team trying to do this, we’d have a spreadsheet on libraries+ maintainers w/ parallel takeover attempts -at least until we had control of #xz Even then we’d keep out other fake GitHub accounts mildly active as contingencies.
Projects with binary files in the test sources are a clear win, hence a focus on compression. But any lib with regression testing of file formats needs those
Also, to be super clear nobody should panic about #XZ as the Postgres developer who found this basically caught it quick enough that almost no businesses or devices will be running the code.
So everybody should be chill about this specific issue as that guy saved everybody’s bacon.
To give an idea of the scale of OpenSSH usage, it’s absolutely huge, it dwarfs RDP by a huge margin (think ten times), and had this survived for a long period of time it would have been unbelievably bad.
@GossiTheDog but we shoud maybe panic at the number of other projects that may have been infiltrated by this actor under other names and have never beeen noticed?
The sshd backdoor in #XZ is just way beyond my technical ability. There’s so much there, I imagine more than a few conference talks are going to be submitted for it.
My amateur hour view is it’s really well put together (eg you can only execute commands if you have a private key that only the attacker has) and appears to allow remote removal of the backdoor, too. There’s a whole bunch of features which I’m too dumb to get.
Also for me, performance isn’t that bad - I wouldn’t have noticed it.
Kinda interesting - a change made by the threat actor has ended up in Windows OS. Redmond bundled libarchive into the OS, which the TA had been tinkering with.
I gotta say, the open source community response to the XZ issue and auditing code changes for the specific threat actor has been incredibly good overall.
4 days since XZ backdoor became public knowledge and most major Linux AV and EDR security vendors still have zero detections.. they haven’t even set the static file hashes as malicious.
Can’t wait for all the vendor blogs in a week saying they fully protect against the threat. 👍
.@amlw wrote a great proof of concept for #XZ to allow code execution via ssh.
Very important note: it doesn’t work in the wild as you need the private key, which only the threat actor(s) have. But you can create your own for exploiting your own servers.
If you use Microsoft Vulnerability Management, it is false positiving on CVE-2024-3094 aka #XZ backdoor - it is picking up the Cygwin version of XZ as vuln on Windows systems.
The Cygwin packages predate the backdoor and it doesn’t impact Windows, also the file it flags isn’t the backdoor but lzmadec.exe
Re #XZ attacker - the known threat actor account made various changes across multiple open source projects and documentation.
Library maintainers should not look at those changes in isolation of just that line change, or assume the threat actor only became malicious later. Assume they are very well resourced and acting with broad objectives.
In at least one case they made an existing unknown vulnerability exploitable, and we know they were socially engineering the XZ maintainer years ago.
‘They’ are very likely a multi million dollar operation - see also just the shell script analysis, before you even get to the backdoor (which is much more nuts) https://research.swtch.com/xz-script
The actual SSH backdoor is cryptographically signed so only the threat actor can use it. If you work in threat intelligence and write “foreign” intelligence agency, you might want to look at your bias training.
@GossiTheDog I am amazed how they took all that time to prepare and effort to hide all this, and then impatiently started pushing the distros in such a suspicious way.
@GossiTheDog That'd blow up systems that haven't downgraded though, no? Detection would be nice, but would risk flooding analysts with alerts they can't do anything about 🤔 (not working on the EDR space, extrapolating)
@GossiTheDog thinking of the “one death is a tragedy, a thousand is a statistic” quote here, but I don’t necessarily disagree. Just thinking that if you have 1000s of vulnerable hosts, getting flooded by alerts wouldn’t be very useful. With that said, I never was on the receiving end so I’m mostly guessing
On the very first day I’ve submitted the static “test” files to ClamAV and VT. ClamAV response was:
Our initial assessment shows that this file is possibly clean. If you provided a description that suggests otherwise, we will further examine the sample & proceed from there.
On VT the detection rate was zero but you can at least add a comment and community ranking there, which I did. Now I see it’s detected by 7/60 and more community ratings were added.
@GossiTheDog somewhere if this is true it would be somewhat funny and hopefully would put a stop to the whole 'opensource vs closed source security' debate
I don't know which is the biggest crime: naming a product or service with a word of the English language, like "upstream", or mentioning that product or service in an article without capitalizing, changing the font, or qualifiying it -- as "software from upstream" rather than "software from Upstream" or "software from the /upstream/ site".
@GossiTheDog@wdormann Several linux distros have already investigated how they're impacted by this (thanks @mgorny and @VoidLinux). Any takes on this from @almalinux and @alpinelinux?
Add comment