I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd, showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol. Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates.
Shows that stubbornly debugging weird issues (that many would probably just ignore as "oh, it's slower now, whatever, it still works well enough..") can pay out big time! :)
(Well, metaphorically at least, though I guess this also increases your worth on the job market)
@AndresFreundTec Gotta add to the noise, many thanks for digging into this. If it wasn't for you, who knows what could've been done here. You're a hero, and I mean it.
@AndresFreundTec I feel both confident and also kind of queasy when saying this: it seems extremely likely that this is not the first time something like this has happened, it's just the first time we have been lucky enough to notice.
Binary artifacts have no business existing in Free Software (or near-binary considering how auditable pre-generated config scripts end-up being). The way it was compromised in this case is almost certain to have happened before and reminds me of the SourceForge malware debacle (so arguably that's another famous example of it happening before).
I"m not sure if many other projects do like Guix and record the checksum of the whole repository so as to ensure reproducibility purely from source.
@lispi314@AndresFreundTec In general this is reasonable, but this there are some clear exceptions for test vectors in cryptographic libraries and compression libraries (which this was).
@glyph@AndresFreundTec In this case the actual malicious vector was the near-binary injected code in the practical binary of the unaudited autotools vomit (always autoreconf) which was then bundled in the actual binary artifact that was the compromised tarballs.
None should have ever been part of the project.
As for the test files, I still think that having a hex dump with comments explaining what flaws particular parts test would be desirable in a lot of cases.
One more aspect that I think emphasizes the number of coincidences that had to come together to find this:
I run a number "buildfarm" instances for automatic testing of postgres. Among them with valgrind. For some other test instance I had used -fno-omit-frame-pointer for some reason I do not remember. A year or so ago I moved all the test instances to a common base configuration, instead of duplicate configurations. I chose to make all of them use -fno-omit-frame-pointer.
Afaict valgrind would not have complained about the payload without -fno-omit-frame-pointer. It was because _get_cpuid() expected the stack frame to look a certain way.
Additionally, I chose to use debian unstable to find possible portability problems earlier. Without that valgrind would have had nothing to complain.
Without having seen the odd complaints in valgrind, I don't think I would have looked deeply enough when seeing the high cpu in sshd below _get_cpuid().
Just to be clear: I didn't mean that I didn't do good - I did. I mean that we got unreasonably lucky here, and that we can't just bank on that going forward.
@AndresFreundTec but there are thousands of devs running all sorts of tests. The question is not "what were the odds that you missed it", but "what were the odds that everybody missed it"...
@HydrePrever@AndresFreundTec The odds that everyone missed it are extremely high. It actually triggered a bunch of test failures, which the attacker explained away as ifunc side effects, and a lot of people simply accepted that explanation.
@AndresFreundTec That was more than just good - you probably stopped a devastating attack on the whole industry. The open source community owes you a huge debt of gratitude. Had that exploit gotten into the wild it would have been awful. Catching it early was an immense win.
I hope Microsoft recognizes how much of a contribution you made to the entire industry.
@AndresFreundTec And a lot of persistence! Reminds me of one of the classics of the industry, Cliff Stoll's Cuckoo's Egg - "Stoll traced the error to an unauthorized user who had apparently used nine seconds of computer time and not paid for it" leading to a german hacker selling content to the KGB - 38 years ago. It is impressive (but uncommon) to see someone paying that level of attention to anomalies these days, with how thick tech stacks have gotten...
The questions at the top of my mind now are: who will fork and continue maintenance of xz? How will we determine that we can trust them? And how will we apply those lessons throughout the larger ecosystem?
@malte@AndresFreundTec There's still a lot of data out there already in xz format, so merely dropping the software would mean that that data becomes unreadable. Dropping it may be an option, but I'm not sure it's the best option.
Add comment