A small blurb from The Guardian on why Andres Freund went looking in the first place.
So how was it spotted? A single Microsoft developer was annoyed that a system was running slowly. That’s it. The developer, Andres Freund, was trying to uncover why a system running a beta version of Debian, a Linux distribution, was lagging when making encrypted connections. That lag was all of half a second, for logins. That’s it: before, it took Freund 0.3s to login, and after, it took 0.8s. That annoyance was enough to cause him to break out the metaphorical spanner and pull his system apart to find the cause of the problem.
At least microsoft is honest enough to admit their software needs protection, unlike apple and unlike most of the people who have made distros of linux. (edit: microsoft is still dishonest about what kind of protection it needs though)
Even though apple lost a class action lawsuit for false advertising over the claim “mac can’t get viruses” they still heavily imply that it doesn’t need an antivirus.
any OS can get infected, it’s just a matter of writing the code and finding a way to deliver it to the system…Now you might be thinking “I’m very careful about what I click on” that’s a good practice to have, but most malware gets delivered through means that don’t require the user to click on anything.
You need an antivirus on every computer you have, linux, android, mac, windows, iOS, all of them. There’s loads of videos on youtube showing off how well or not so well different antivirus programs work for windows and android.
Most of the time, detection also means prevention, but with a whitelisting antivirus, prevention often means that the threat isn’t detected, it was just prevented from running.
A whitelisting application has a list of what it knows it bad AND what it knows in advance to be good.
Anything it can’t identify on the spot is treated as unknown and not allowed to run, not deleted, not quarantined, just blocked from running until the user can upload it to things like virustotal and other services like it to figure out if its safe.
upload it to virustotal, if it wasn’t already known, do a re-scan a few hours later to see if it’s malicious, if it was already known, do a re-scan to see if anything has figured out if its malicious.
which is why I think it’s borderline criminal that most antivirus programs don’t work that way.
A whitelisting application has a list of what it knows it bad AND what it knows in advance to be good.
How would it know this? Is this defined by a person/people? If so, that wouldn’t have mattered. liblzma was known in advance to be good, then the malicious update was added, and people still presumed that it was good.
This wasn’t a case of some random package/program wreaking havoc. It was trusted malicious code.
Also, you’re asking for an antivirus that uploads and uses a sandbox to analyze ALL packages. Good luck with that. (AVs would probably have a hard time detecting malicious build actions, anyways).
Also, you’re asking for an antivirus that uploads and uses a sandbox to analyze ALL packages. Good luck with that. (AVs would probably have a hard time detecting malicious build actions, anyways).
three different antivirus programs already do that. Comodo for example has a built in sandbox to do that.
The scary thing about this is thinking about potential undetected backdoors similar to this existing in the wild. Hopefully the lessons learned from the xz backdoor will help us to prevent similar backdoors in the future.
As mentioned, binary test files makes sense for this utility. In the future though, there should be expected to demonstrate how and why the binary files were constructed in this way, kinda like how encryption algorithms explain how they derived any arbitrary or magic numbers. This would bring more trust and transparency to these files without having to eliminate them.
It’s not uncommon to keep example bad data around for regression to run against, and I imagine that’s not the only example in a compression library, but I’d definitely consider that a level of testing above unittests, and would not include it in the main repo. Tests that verify behavior at run time, either when interacting with the user, integrating with other software or services, or after being packaged, belong elsewhere. In summary, this is lazy.
A repo dedicated to non-unit-test tests would be the best way to go. No need to pollute your main code repo with orders of magnitude more code and junk than the actual application.
That said, from what I understand of the exploit, it could have been avoided by having packaging and testing run in different environments (I could be wrong here, I’ve only given the explanation a cursory look). The tests modified the code that got released. Tests rightly shouldn’t be constrained by other demands (like specific versions of libraries that may be shared between the test and build steps, for example), and the deploy/build step shouldn’t have to work around whatever side effects the tests might create. Containers are easy to spin up.
Keeping them separate helps. Sure, you could do folders on the same repo, but test repos are usually huge compared to code repos (in my experience) and it’s nicer to work with a repo that keeps its focus tight.
It’s comically dumb to assume all tests are equal and should absolutely live in the same repo as the code they test, when writing tests that function multiple codebases is trivial, necessary, and ubiquitous.
It’s also easier to work if one simple git command can get everything you need. There is a good case for a bigger nono-repo. It should be easy to debug tests on all levels else it’s hard to fix issues that the bigger tests find. Many new changes in git make the downsides of a bigger repo less hurtful and the gains now start to outweigh the losses of a bigger repo.
I would say yes and no, but yes the clone command can do it. But branching and CI get a bit more complicated. Pushing and reviewing changes gets more complicated to get the overview. If the functionality and especially the release cycle is different the submodules still have great values. As always your product and repo structure is a mix of different considerations and always a compromise. I think the additions in git the last years have made the previous really bad pain points with bigger repos less annoying. So that I now see more situations it works well.
I always recommend keeping all testing in the same repo as the code that affects the tests. It keeps tracking changes in functionality easier, needing to coordinate commits, merging, and branches in more than one repo is a bigger cognitive load.
I agree that in most cases it’s more of an E2E or integratiuon test, not sure of the need to split into different repo, and well in the end I’m not sure that would have made any big protection anyhow.
Is it, really? If the whole point of the library is dealing with binary files, how are you even going to have automated tests of the library?
The scary thing is that there is people still using autotools, or any other hyper-complicated build system in which this is easy to hide because who the hell cares about learning about Makefiles, autoconf, automake, M4 and shell scripting at once to compile a few C files. I think hiding this in any other build system would have been definitely harder. Check this mess:
<span style="color:#323232;"> dnl Define somedir_c_make.
</span><span style="color:#323232;"> [$1]_c_make=`printf '%sn' "$[$1]_c" | sed -e "$gl_sed_escape_for_make_1" -e "$gl_sed_escape_for_make_2" | tr -d "$gl_tr_cr"`
</span><span style="color:#323232;"> dnl Use the substituted somedir variable, when possible, so that the user
</span><span style="color:#323232;"> dnl may adjust somedir a posteriori when there are no special characters.
</span><span style="color:#323232;"> if test "$[$1]_c_make" = '"'"${gl_final_[$1]}"'"'; then
</span><span style="color:#323232;"> [$1]_c_make='"$([$1])"'
</span><span style="color:#323232;"> fi
</span><span style="color:#323232;"> if test "x$gl_am_configmake" != "x"; then
</span><span style="color:#323232;"> gl_[$1]_config='sed "rn" $gl_am_configmake | eval $gl_path_map | $gl_[$1]_prefix -d 2>/dev/null'
</span><span style="color:#323232;"> else
</span><span style="color:#323232;"> gl_[$1]_config=''
</span><span style="color:#323232;"> fi
</span>
Though unfortunately (or I guess for most use-cases fortunately) you can’t find the malicious m4/build-to-host.m4 file on there afaik. The best way to find that now, should you really want to, is by looking through the commit history of the salsa.debian.org/debian/xz-utils repository which is, as far as I understand it, the repository that the debian packages are built from and consequently also what the compromised packages were built from.
I think going forward we need to look at packages with a single or few maintainers as target candidates. Especially if they are as widespread as this one was.
In addition I think security needs to be a higher priority too, no more patching fuzzers to allow that one program to compile. Fix the program.
There’s gotta be a better way to verify programs then just what the devs do. For example patching the fuzzer, that should be seen as a clear separation of duties problem.
That constant issue of low Dev/high use dependencies is awful and no one I’ve met on the business end can seem to figure out that need to support those kind of people or accept, what should frankly be, legal liability for what goes wrong. This isn’t news its just a cover song. And its not an open source problem, its just a software problem. (
In the words of the devs in that security email, and I’m paraphrasing -
“Lots of people giving next steps, not a lot people lending a hand.”
I say this as a person not lending a hand. This stuff over my head and outside my industry knowledge and experience, even after I spent the whole weekend piecing everything together.
You are right, as you note this requires a set of skills that many don’t possess.
I have been looking for ways I can help going forward too where time permits. I was just thinking having a list of possible targets would be helpful as we could crowdsource the effort on gitlab or something.
I know the folks in the lists are up to their necks going through this and they will communicate to us in good time when the investigations have concluded.
no more patching fuzzers to allow that one program to compile. Fix the program
Agreed.
Remember Debian’s OpenSSL fiasco? The one that affected all the other derivatives as well, including Ubuntu.
It all started because OpenSSL did add to the entropy pool a bunch uninitialized memory and the PID. Who the hell relies on uninitialized memory ever? The Debian maintainer wanted to fix Valgrind errors, and submitted a patch. It wasn’t properly reviewed, nor accepted in OpenSSL. The maintainer added it to the Debian package patch, and then everything after that is history.
Everyone blamed Debian “because it only happened there”, and definitely mistakes were done on that side, but I surely blame much more the OpenSSL developers.
I’d have to dig it, but I think it said that it added the PID and the uninitialized memory to add a bit more data to the entropy pool in a cheap way. I honestly don’t get how that additional data can be helpful. To me it’s the very opposite. The PID and the undefined memory are not as good quality as good randomness. So, even without Debian’s intervention, it was a bad idea. The undefined memory triggered valgrind, and after Debian’s patch, if it weren’t because of the PID, all keys would have been reduced to 0 randomness, which would have probably raised the alarm much sooner.
This has always been the case. Maybe I work in a unique field but we spend a lot of time duplicating functionality from open source and not linking to it directly for specifically this reason, at least in some cases. It’s a good compromise between rolling your own software and doing a formal security audit. Plus you develop institutional knowledge for that area.
And yes, we always contribute code back where we can.
If this was done by multiple people, I’m sure the person that designed this delivery mechanism is really annoyed with the person that made the sloppy payload, since that made it all get detected right away.
I like to imagine this was thought up by some ambitious product manager who enthusiastically pitched this idea during their first week on the job.
Then they carefully and meticulously implemented their plan over 3 years, always promising the executives it would be a huge pay off. Then the product manager saw the writing on the wall that this project was gonna fail. Then they bailed while they could and got a better position at a different company.
The new product manager overseeing this project didn't care about it at all. New PM said fuck it and shipped the exploit before it was ready so the team could focus their work on a new project that would make new PM look good.
The new project will be ready in just 6-12 months, and it is totally going to disrupt the industry!
I see a dark room of shady, hoody-wearing, code-projected-on-their-faces, typing-on-two-keyboards-at-once 90’s movie style hackers. The tables are littered with empty energy drink cans and empty pill bottles.
A man walks in. Smoking a thin cigarette, covered in tattoos and dressed in the flashiest interpretation of “Yakuza Gangster” imaginable, he grunts with disgust and mutters something in Japanese as he throws the cigarette to the floor, grinding it into the carpet with his thousand dollar shoes.
Flipping on the lights with an angry flourish, he yells at the room to gather for standup.
But it would be nice to have a similar digram like OP’s to understand how exactly it does the RCE and implements the SSH backdoor. If we understand how, maybe we can take measures to prevent similar exploits in the future.
I think ideas about prevention should be more concerned with the social engineering aspect of this attack. The code itself is certainly cleverly hidden, but any bad actor who gains the kind of access as Jia did could likely pull off something similar without duplicating their specific method or technique.
I imagine if this attacker wasn’t in a rush to get the backdoor into the upcoming Debian and Fedora stable releases he would have been able to notice and correct the increased CPU usage tell and remain undetected.
Under the right circumstances this interference could potentially enable a malicious actor to break sshd authentication and gain unauthorized access to the entire system remotely.
—Wikipedia, sourced to RedHat
Of course, the authentication bypass allows remote code execution.
Basically, if you have a patched SSH client with the right ED448 key you can have the gigged sshd on the other side run whatever commands you want. The demo just does id > /tmp/.xz but it could be whatever command you want.
I am not a security expert, but the scenario they describe sounds exactly like authentication bypass to a layman like me.
According to www.youtube.com/watch?v=jqjtNDtbDNI the software installs a malicious library that overwrite the signature verification function of ssh.
I was wondering if the bypass function was designed to be slightly less resource intensive, it probably won’t be discovered and will be shipped to production.
Also I have mixed feeling about dynamic linking, on the one hand, it allows projects like harden malloc to easily integrate into the system, on the other hand, it also enables the attacker to hijack the library in a similar fashion.
EDIT: This is a remote code execution exploit, not authentication bypass. The payload is sent as an authentication message and will be executed by the compromised authentication function.
This means:
the payload will be executed as root, since sshd run as root.
the payload will leave no trace in login log.
So this is much worse than ssh authentication bypass.
Authentication bypass should give you interactive access. “I’m in” like. Remote code execution only allows you to run a command, without permanent access. You can use some RCE vulnerabilities to bypass authentication, but not all.
Yeah, but the malicious code replaces the ssh signature verification function to let it allow a specific signature. Hence attacker, with the key, can ssh into any system without proper authentication by ssh.
This kind of describes authentication by-pass, not just remote code execution…
EDIT: it is remote code execution, see the edit of parent comment.
Add comment