Doing software engineering (well-ish) is pretty hard to imagine without working in version control, which most of the time means git. In a practical setup of git there’s the question of how do I get access to the code it stores — how do I “check things out”? — and optionally how can others verify that it was indeed me who did the changes — how do I “sign” my commits? Recently I’ve changed my mind about what’s a good combination for these two aspects, and what tools am I using to do them.
Access Options
In broad terms git repositories can be checked out either though the HTTP protocol, or through the SSH protocol. Both have pros and cons.
Having two-factor authentication (2FA) made the HTTP access more secure but also more setup (no more direct username/password usage, rather needing to create extra access keys used in place of passwords). Credentials were still in plain text (as far as I know) on the machine in some git config files.
The SSH setup was in some sense more practical one (creating keys on your own machine, and just passing in the public key portion), though there were still secrets in plain text on my machine (as I don’t think the majority of people used password-protected SSH keys, due to their user experience). This is what I’ve used for years: add a new SSH key for a new machine that I’m working on, check code out through ssh+git, and work away.
When I’ve recently came across the git-credential-manager tool that supposed to make HTTP access nicer (for various git servers and services), and get rid of plain text secrets. Of course this is not the first or only one of the tools that does git credentials, but being made by GitHub, it had some more clout. This made me re-evaulate what options do I have for SSH as well for similar security improvements.
Thus I’ve found that both 1Password and KeePassXC (the two main password managers I use) have ssh-agent integration, and thus can store SSH keys + give access to them as needed. No more plain text (or password protected) private keys on disk with these either!
Now it seems there are two good, new options to evaulate, and for the full picture I looked at how the code signing options work in this context as well.
The GPG setup is well established, and also links my commits to my identities used elsewhere (e.g. signed emails sent to mailing lists that care about it, with the key linked from this site’s frontpage). This of course is not always needed or desired, but it decouples the identity from the code hosting platform. There’s some serious downsides as well, though: GPG signing keys are not supposed to be numerous (just a single one), and thus if I use multiple machines to work on, I will have to take my private keys with me between machines, for example making copies of them. Or if not making copies, then have them on hardware keys (that have other problems with backups and all that, if I got it right the last time I tried to understand the process).
The SSH key commit signing is much newer (need git version at least 2.34), but it’s also simpler: add a key to my git hosting service, sign commits with that key, and thus the service can match things up and show that match. I can add as many keys as machines I’m working on if needed, no need to transfer or copy keys between machines, and I can also choose use some keys for login only or code signing only.
A third party trying to verify these signatures, though, would need to get the keys from the hosting service (I’d find it surprising if people would distribute their commit signing keys out of band the same way as they do with GPG public keys, since there are likely more of them). Hence it git hosting services will need to make the user’s keys available (as they do at the relevant username.keys URLs, e.g. mine on GitHub and GitLab).
Also can’t forget to add the relevant keys to the list of allowed signing keys locally, and all the other relevant setup (see e.g. the GitHub and GitLab docs). There are a bit too many places to update, but it’s mostly set-it-and-forget-it. After that, once started to sign commits, adding the --show-signatures flag to the commands that support it (git log, git show for example), should show the signatures.
My Winning Combo
Looking at the opions above, there’s a matrix of options that we can use, and here’s what I think about them:
GPG signature
SSH key signature
Git Credential Helper
Extra setup
Simpler
SSH clone
The usual
Most convenience 👍
Convenience matrix of Git access (rows) and commit signing (columns) optionsReally, where I want to be is just SSH keys for everything, even if they are imperfect, but they have the most number of puzzle pieces to fit.
SSH Key Security
While previously SSH keys were really just held as files in your ~/.ssh folder, most likely, recently I’ve found (tada!) that the password managers I use can also store & serve SSH keys: see in particular 1Password’s SSH documentation and KeePassXC docs (scroll to SSH Agent Integration on that page), though I’m sure other password managers can do this too.
1Password
The two password managers listed above handle things quite similarly. 1Password is a bit less hands-on, though, the default settings work pretty well.
With this things generally work, and relatively easy to reason about things. When things are less clear, it might be just a debug check-away away from seeing the keys added to this alternate agent:
KeePassXC, being open source, is my preferred solution of the two, though unsurprisingly it’s the more awkward one to set up. The main differences from 1Password include:
needing to generate the keys externally to the password manager (rather than having built-in ssh keygen) – this is a con on usability but a strong pro on basing security on the established tool, rather than potentially questionably reimplement it
uses the main SSH agent, so no extra setup is necessary in most of the tools – this is a potential pro on usability for configurations, but a potential con that the worflow and config of loading keys into the agent needs a bit more understanding to be both ergonomic and safe to one’s level of paranoia
the key use confirmation defaults to “ok” on pressing Enter on the pop-up (rather than Cancel) – this is a pro on usability, but con on “failing open” rather than closed
It’s still a pretty simple workflow, and it’s quite interesting to see how many things KeePassXC learned to do as well.
Experience
Thinking about the various threat models to my SSH crendentials, this setup adds one more layer to the defence in depth, and it does feel more relaxed already (relaxed from a point of stress I didn’t quite know I had before).
Picking the SSH key based login and signing feels like using the most appropriate tech for the moment, and there are still knobs for people to adapt it to their security levels (different SSH keys for login and signing, passwords on the keys themselves, etc…)
This setup works very well when I want to be notified whenever a tool’s using the SSH key so it would be more obvious if a stray process is trying, say exiltrate the keys. On the other hand this breaks down when git itself is running background processes, such as git-maintenance, so that’s not something that I could use here. So far out of (literally) thousands of codebases & repos I’ve used that maintenance setup exactly once, for convenience. For me it is not a major loss, then.
The one bit that feels a step backwards is that having the SSH keys in the password manager and carrying it around counteracts the “separate key for each system” arrangement. This might just be part of getting used to new processes, and not an actual downside.
Further Thoughts
In cybersecurity yesterday’s best practices might be inadecvate today and “last week’s” practices might be outright dangerous… Gonna keep revisiting this setup more broadly and in terms of details, as I learn more.
It’s a good question why even do code signing (besides having a “verified” check mark, which alone doesn’t mean much if not part of a verification process), though this needs some more space to unpack. For the time being I’ll assume that signing is better than not signing, if nothing else than as a forward looking prep for better audit processes down the line.
There’s really a question around having too many things in a single password manager: nowadays it can be the complete “royal flush” of password, TOTP, SSH key, recovery codes, passkey… and likely more bits that I might not be using yet? This does make me uneasy, and likely a scale on which usability and security will adjust over time (such as. bundling and unbundling various cybersecurity aspects).
I might also actually misunderstand various things above, if so, I’d be very keen to hear, just drop me a line!
During lunch a friend mentioned that you can just supply a HTTP URL to vim on the command line and it would use curl to download that resource and allow you to edit the content. I jokingly asked whether if you enter :w it would then issue a HTTP POST back to the origin which is of course ridiculous.
pyinfra turns Python code into shell commands and runs them on your servers. Execute ad-hoc commands and write declarative operations. Target SSH servers, local machine and Docker containers. Fast and scales from one server to thousands.
Endlessh is an #SSH tarpit that very slowly sends an endless, random SSH banner. It keeps SSH clients locked up for hours or even days at a time. The purpose is to put your real SSH server on another port and then let the script kiddies get stuck in this tarpit instead of bothering a real server. https://github.com/skeeto/endlessh
Forgot to toot, I had tried to use #SSH#X11Forwarding to run graphic applications on my servers (#raspberrypi , and even a broken-screen #pinephone with Wayland !)
After some configuration all of them work. Yay I can now run x11 apps on headless machines.
Mettez à jour Putty, Filezilla, TortoiseGit et WinSCP si vous utilisez des clés SSH pour vous connecter à certains serveurs (dans l'idéal faites aussi la mise à jour si vous n'êtes pas concerné). https://www.programmez.com/node/36392#Security#Hack#SSH
Static sites, RSS feed to email digest, reverse ssh tunnels (for exposing local socket with public domain name), blog engine, pastebin and couple more.
We've released #PuTTY version 0.81. This is a SECURITY UPDATE, fixing a #vulnerability in ECDSA signing for #SSH.
If you've used a 521-bit ECDSA key (ecdsa-sha2-nistp521) with any previous version of PuTTY, consider it compromised! Generate a new key pair, and remove the old public key from authorized_keys files.
Other key types are not affected, even other sizes of ECDSA. In particular, Ed25519 is fine.
We're excited to be joining an ever-growing number of signatories that committ to
✅ working toward making openness the default for research information;
✅ working with services and systems that support and enable open research information;
✅ supporting the sustainability of infrastructures for open research information;
✅ and supporting collective action to accelerate the transition to openness of research information.
In our context, we work hard to make that happen by providing an #opensource platform for small, scholar-led, and university publishers to manage and disseminate fully open #CC0#metadata for #OAbooks - which is of particular relevance to the #Humanities#SSH!
As the saying goes, "Real BOFH use tar and rsync!"
The blog article is an excellent treatment of using tar along with SSH to effect a reliable backup plan and schedule.
Another couple of great fav GoTo solutions of mine have always been Duplicity and Duply for those not comfortable rolling their own scripts w/SSH, tar, and/or rsync :batman:
Thank you very much for sharing this @nixCraft !!!
If you've ever looked at SSH server logs you know what I'm about to say: Any SSH server connected to the public Internet is getting bombarded by constant attempts to log in. Not just a few of them. A lot of them. Sometimes even dozens per second. And this problem is not going away; it is, in fact, getting worse. And attackers' behavior is changing.
The graph attached to this post shows the number of attempted SSH logins per day to one of @cloudlab s clusters over a four-year period. It peaks at about 3.4 million login attempts per day.
This is part of a study we did on our production system, using logs of more than 640 million login attempts, covering more than 1,500 hosts on our side and observing more than 840 thousand incoming IP addresses.
A paper presenting our analysis and a new, highly effective means to block SSH brute force attacks ("Where The Wild Things Are: Brute-Force SSH Attacks In The Wild And How To Stop Them") will be presented next week at #NSDI24 by @sachindhke . The full paper is at https://www.flux.utah.edu/paper/singh-nsdi24
Wir sind dieses Wochenende nur durch unglaubliches Glück und extrem knapp an wohl einer der grössten Katastrophen rund um die globale IT-Sicherheit vorbeigeschrammt.
Phuh! Doch — was ist eigentlich passiert? Wie konnte das überhaupt geschehen? Und was können (und müssen) wir tun, um dies zukünftig zu vermeiden?
#CyberSecurity#OpenSource#Malware#SSH#XZUtils: "So what was the malware discovered by Freund designed to do? Basically to break the authentication process that makes SSH secure and thereby create a backdoor that would enable an intruder remotely to gain unauthorised access to the entire system. Since SSH is a vital tool for the safe operation of a networked world, anything that undermines it is really bad news – which is why the cybersecurity world has been on high alert in the past week. Those running the different flavours of Linux that are in use across the world have been alerted to the dangers posed by the two rogue updates.
In some ways, the story of how the malware got into the updates is even more instructive. XZ Utils is open-source software, ie software with source code that anyone can inspect, modify and enhance. Much open source is written and maintained by small teams of programmers, and in many case by a single individual. In XZ Utils, that individual for years has been Lasse Collin, who has been with the project since its inception. Until recently he was the person who had been assembling and distributing the updates of the software." https://www.theguardian.com/commentisfree/2024/apr/06/xz-utils-linux-malware-open-source-software-cyber-attack-andres-freund