I know the next 3-7 days will be filled with exaggeration and doomsday talk, but IMHO the #xz backdoor, though seemingly meticulously planned for a long time, failed miserably as it was caught at a stage where it wasn't widely deployed but only in testing/prerelease distros. Yes, it made it quite far in the supply chain but it ultimately failed. The mess is being cleaned up, no cases of actual use of the exploit in the wild are known thus far. The immune system of FOSS has worked. Again.
The back door was discovered by serendipity. It could easily have gone unnoticed, if not for one person randomly noticing that logging in via ssh had gotten slightly slower, and then actually bothering to investigate why.
We're relying on luck to detect these attacks, and luck eventually runs out.
@jwildeboer not unfair but I also think it deserves not to be undersold.
IMHO this is definitely a structural problem, it has happened before and will happen again; people simply don't care about security enough, this doesn't just pertain to Linux, it's just as likely to happen with companies with beancounters that try and spend as little as possible on security as it's far easier to sell new features or even a UI refresh than it is to sell security.
I wonder how this can be circumvented and detected much earlier?n
In this case, building the source from the repo and compare it to the release tarball would have detected the additional m4 file. Maybe distributions need to start checking for these kind of indicators of compromise. @jwildeboer
@kirschwipfel@jwildeboer it's always by chance. All the checks, processes etc. only affect the probability of catching the backdoor. You'll never get to 100%.
This a bit reminds the story of a backdoor in Unix C compiler.
I disagree. All these checkers are part of the process, while "some developer of a totally unrelated project does perf monitoring" is not.
Both the checkers and Andreas could have missed detecting the backdoor. Its been pure luck Andreas did perf monitoring, has discovered anomalies and followed them.
Calling this "open source worked" is wrong: OSS made it easy to track it down after the incident already happened. It did not prevent the incident. @aurisc4@jwildeboer
@kirschwipfel@jwildeboer It's actually kinda shocking to me that debian and redhat allow release tarballs to include things that weren't present in the source, that effectively makes them a much easier target as now ONLY the devs on those distros would ever be able to see the actual exploit.
It's actually a fairly common thing that you have to execute ./autogen.sh before being able to run ./configure when building a software package from a git checkout, whilst release tarballs will usually come with a working ./configure script right out the box (no pun intended).
Also, if the two diverge, package maintainers will usually prefer release tarballs because oftentimes only those tarballs have proper PGP signatures, whilst git commits and tags are often unsigned.
@srslypascal that's kinda my point though, in a security focused distro with PAID support like RedHat you would expect everything to be checked out from that branch and built from their in-house build system, so the build actually matches the one that's actually being widely tested by developers and other distros. if you're allowing unchecked files in release tarballs or only relying on in-house maintainers even for external packages that seems like a big miss.
@jwildeboer Something I think about is: This was a fail and hence was noticed, but are there other successful attempts that weren't caught? So we have to care about developers mental health (so they don't give up their project to someone eager to take ownership carelessly), good review processes (so this attempts get caught quickly) and no paranoia but carefulness.
@jwildeboer while what you say is indeed true, this was way too close for comfort. It also makes me wonder how many others are out there that hasn't been detected.
@jwildeboer I hope this brings enough attention to these underfunded projects and brings about a change to make a more sustainable development model for these kinds of projects.
@Aaron You can be sure that many, many people are going through this as we speak. And checking code is always better than wasting time on public flamewars ;)
I will simply ignore the whole public story for a bit and instead follow the messages on the technical channels where the actual analysis, impact assessment and consequences are discussed. The full story on how this happened and who is behind it are still mostly unknown. Speculations are not helpful. Fingerpointing and waving conspiracy theories around are just not my cup of tea. And they can easily damage the wrong persons.
@jwildeboer wonder if anyone is looking to see if there any other build commits that have the same sig of injecting code in the way this was done. if it's a state actor or a big commercial actor will be likely not to be the only attempt
Add comment