As a longtime provider of services in one form or another since the late 80's and early 90's, I felt the pain of having to write out the following blog post/update.
Drew is an opinionated perfectionist with an attention to detail and his perspective that chafes some, endears others, and deservedly, receives the respect earned when someone strives toward par excellence for those for whom they provide services for.
I have some differing set of conclusions from my understanding of what he laments as the ordeal he's been through in the past year, like, "why would anyone consider a carrier besides DHL for international overseas shipments?" Also, I fail to see the logic in moving his entire infra from the U.S. (where there are many affordable top-tier carrier hotels - aka datacenters) to Amsterdam, which also has fine facilities and maybe it is because of privacy concerns which depending on what those are, may indeed be quite valid from my perspective.
But not having IPv6 fully deployed (as a result of datacenter choice?) is puzzling, although almost inconsequential operationally, in production, ... Almost.
Considering I've always looked directly at the carriers themselves, used my own delegated IP infrastructure for core operations, I tend to look at a datacenter as three things:
Electricity
Fail-over electricity (Generators)
Air conditioning
Most folks rent a rack that comes with transit, I ask how much the XC is - I can find, mix, and pick my transit providers. I just wanna know that my shit is secure in a suite or cage behind locked cabinets that I personally have 24/7 access to at anytime (even though I'll rarely do so) and have 24/7 remote hands to swap drives, hot-pluggable power supplies and plug cables into the designated ports I specify, etc. Those things typically come w/zero cost.
For DDoS'ing, I do like to outsource this as part of a package, and I'm open to any offers of included transit/XC and want to know how much each additional 20A of electricity cost me each month in addition to the rack fees. Putting the onerous of protecting my customers from a good DDoS'ing on someone else like my upstream takes a lot of worry away.
Shipping machinery though, that's a bit distinct too, I've been burned a few times domestically, although always recovered my *tangible costs - time? well, I've lost a couple of customers because their infra was lost or damaged in transit, but insurance is important - Drew had that. What I'm really wondering though, is who besides DHL would you even trust to ship servers over the Atlantic Ocean?
That's a cost I would not consider skimping on - A girl I almost married worked for DHL for over 20 years and they'll cut a check at the drop of a hat, which might have worked out well for Drew considering these were old boxes ready for retirement anyway and the replacement cost (new stuffs) is what you insure for.
Anyway, I've really admired much of what Drew has done over the years, was cheerleading for him as he migrated from full time paycheck person to finally being able to announce that he "thinks" he can make enough money for a living by devoting himself full time to FOSS with his fledgling SourceHut.
Yah, sometimes his head swelled up pretty big, making it hard to fit through doorways, and I've butted heads with him here and there on technical matters only, but have always respected him, and in truth, he was never not correct even if his way was the wrong way, or there was simply a better way - usually those were matters of opinion coz there's more than six ways to Sunday to skin a cat.
Anyway, he's been kicked in the balls really hard, which if you know much of him, must have been really hard to lay all of that out in some manner of detail (He's almost always brutally transparent). For that, and moreover for getting right back up after being knocked down (maybe by da man?), I applaud his candidness. His devotion to those of you reading this that may have free repos at SourceHut, and I'm also encouraging everyone to kick in at least a few bucks - fuck that dumb app that you don't need, let alone pay $2.95 for the exclusive right to be tracked - I urge you with all FOSSiness in mind... Give it a read, and send him whatev, ... I guarantee it will come back to you tenfold.
Drew is a consummate FOSS warrior, do it for yourself, please - Five bucks, fifty bucks, heck, whatever isn't going to cut into your budget for porterhouse steak this weekend would be nice.
And it will make you feel good too.
Full disclosure: I'm not getting shit from this article. Drew and I only converse occasionally and usually it is to disagree - some folks are just good coz of what's in their heart, their commitment to the community, and whether you're a fan or not doing this for him really is doing this for yourself and everyone else in the FOSS world.
I'm a huge fan of "git log --oneline --graph" but I've noticed merge commits are indented, which doesn't seem correct to me.
In the attached screenshots, I've highlighted the extra spaces I don't think should be there in the first and in the second I've modified the indenting to what I expect to see.
interesting to hear somewhere that GitHub is working to integrate ActivityPub in some form or fashion... Sounds very cool. Both excel at being distributed, etc etc.
Of course, searching for "github activitypub" just brings me to every ActivityPub project page and not what I was actually looking for 😂
Is it just me or do @Codeberg have a lot of issues these days? I can't reach their website at the moment and this is far from the only time I can't reach codeberg.org.
Git pathspec glob pattern starting with **/ should match in all directories. However, it does not appear to match in the top level directory, only subdirectories.
For example, 'git ls-files **/Makefile' does not find root Makefile.
So, to find all Makefiles, possibly with suffixes, you'd have to use 'git ls-files Makefile* */Makefile', which is a bit tedious.
Am I missing some magic syntax to achieve what I want?
Is there a #git feature I can enable to hit me on the head with a Looney Tunes-style mallet if I run git checkout -b and it doesn't successful create and and switch branch?
So I don't ignore the error and accidentally commit to master.
If you're wondering what #git forge to host your #opensource code project on, the right answer is probably going to be #forgejo. If the right answer was anything else, you probably wouldn't be wondering about it.
cursed #git feature of the day: git-archive can substitute things in tarballs, which can be used to include the commit hash or even tag (!) inside a file in a source tarball.
Talking to someone about git's UI, and they compared it to vim and GUI IDEs.
When replying with how vi was basically a GUI of its time over more CLI editing with ed/ex ...
it struck me that it is perhaps glaring that we don't have a "vgit": A more visual/TUI tool that supplanted and erased git from memory apart from the "git compatibility mode" still available in "vgit".
I may be off here, but is this emblematic of the cultish worship of unix tooling in the "linux" era?
The regpg program is a thin wrapper around gpg for looking after secrets that need to be stored encrypted in a version control system (so you don't have to trust the VCS server) and decrypted when your configuration management system deploys them to servers.
"'[…] #git was created as a tool to unblock future #Linux#kernel releases — not intended as a global reinvention of all source code management; Linus’s comments highlight that he explicitly saw source code management as the domain of other tools that would then interface with git. […]'"
If you put your Xcode project in a folder that already has a .git repo in it - you likely will be doing commits to not where you think you are. 😱🤦🏻♂️ #git#xcode
Doing software engineering (well-ish) is pretty hard to imagine without working in version control, which most of the time means git. In a practical setup of git there’s the question of how do I get access to the code it stores — how do I “check things out”? — and optionally how can others verify that it was indeed me who did the changes — how do I “sign” my commits? Recently I’ve changed my mind about what’s a good combination for these two aspects, and what tools am I using to do them.
Access Options
In broad terms git repositories can be checked out either though the HTTP protocol, or through the SSH protocol. Both have pros and cons.
Having two-factor authentication (2FA) made the HTTP access more secure but also more setup (no more direct username/password usage, rather needing to create extra access keys used in place of passwords). Credentials were still in plain text (as far as I know) on the machine in some git config files.
The SSH setup was in some sense more practical one (creating keys on your own machine, and just passing in the public key portion), though there were still secrets in plain text on my machine (as I don’t think the majority of people used password-protected SSH keys, due to their user experience). This is what I’ve used for years: add a new SSH key for a new machine that I’m working on, check code out through ssh+git, and work away.
When I’ve recently came across the git-credential-manager tool that supposed to make HTTP access nicer (for various git servers and services), and get rid of plain text secrets. Of course this is not the first or only one of the tools that does git credentials, but being made by GitHub, it had some more clout. This made me re-evaulate what options do I have for SSH as well for similar security improvements.
Thus I’ve found that both 1Password and KeePassXC (the two main password managers I use) have ssh-agent integration, and thus can store SSH keys + give access to them as needed. No more plain text (or password protected) private keys on disk with these either!
Now it seems there are two good, new options to evaulate, and for the full picture I looked at how the code signing options work in this context as well.
The GPG setup is well established, and also links my commits to my identities used elsewhere (e.g. signed emails sent to mailing lists that care about it, with the key linked from this site’s frontpage). This of course is not always needed or desired, but it decouples the identity from the code hosting platform. There’s some serious downsides as well, though: GPG signing keys are not supposed to be numerous (just a single one), and thus if I use multiple machines to work on, I will have to take my private keys with me between machines, for example making copies of them. Or if not making copies, then have them on hardware keys (that have other problems with backups and all that, if I got it right the last time I tried to understand the process).
The SSH key commit signing is much newer (need git version at least 2.34), but it’s also simpler: add a key to my git hosting service, sign commits with that key, and thus the service can match things up and show that match. I can add as many keys as machines I’m working on if needed, no need to transfer or copy keys between machines, and I can also choose use some keys for login only or code signing only.
A third party trying to verify these signatures, though, would need to get the keys from the hosting service (I’d find it surprising if people would distribute their commit signing keys out of band the same way as they do with GPG public keys, since there are likely more of them). Hence it git hosting services will need to make the user’s keys available (as they do at the relevant username.keys URLs, e.g. mine on GitHub and GitLab).
Also can’t forget to add the relevant keys to the list of allowed signing keys locally, and all the other relevant setup (see e.g. the GitHub and GitLab docs). There are a bit too many places to update, but it’s mostly set-it-and-forget-it. After that, once started to sign commits, adding the --show-signatures flag to the commands that support it (git log, git show for example), should show the signatures.
My Winning Combo
Looking at the opions above, there’s a matrix of options that we can use, and here’s what I think about them:
GPG signature
SSH key signature
Git Credential Helper
Extra setup
Simpler
SSH clone
The usual
Most convenience 👍
Convenience matrix of Git access (rows) and commit signing (columns) optionsReally, where I want to be is just SSH keys for everything, even if they are imperfect, but they have the most number of puzzle pieces to fit.
SSH Key Security
While previously SSH keys were really just held as files in your ~/.ssh folder, most likely, recently I’ve found (tada!) that the password managers I use can also store & serve SSH keys: see in particular 1Password’s SSH documentation and KeePassXC docs (scroll to SSH Agent Integration on that page), though I’m sure other password managers can do this too.
1Password
The two password managers listed above handle things quite similarly. 1Password is a bit less hands-on, though, the default settings work pretty well.
With this things generally work, and relatively easy to reason about things. When things are less clear, it might be just a debug check-away away from seeing the keys added to this alternate agent:
KeePassXC, being open source, is my preferred solution of the two, though unsurprisingly it’s the more awkward one to set up. The main differences from 1Password include:
needing to generate the keys externally to the password manager (rather than having built-in ssh keygen) – this is a con on usability but a strong pro on basing security on the established tool, rather than potentially questionably reimplement it
uses the main SSH agent, so no extra setup is necessary in most of the tools – this is a potential pro on usability for configurations, but a potential con that the worflow and config of loading keys into the agent needs a bit more understanding to be both ergonomic and safe to one’s level of paranoia
the key use confirmation defaults to “ok” on pressing Enter on the pop-up (rather than Cancel) – this is a pro on usability, but con on “failing open” rather than closed
It’s still a pretty simple workflow, and it’s quite interesting to see how many things KeePassXC learned to do as well.
Experience
Thinking about the various threat models to my SSH crendentials, this setup adds one more layer to the defence in depth, and it does feel more relaxed already (relaxed from a point of stress I didn’t quite know I had before).
Picking the SSH key based login and signing feels like using the most appropriate tech for the moment, and there are still knobs for people to adapt it to their security levels (different SSH keys for login and signing, passwords on the keys themselves, etc…)
This setup works very well when I want to be notified whenever a tool’s using the SSH key so it would be more obvious if a stray process is trying, say exiltrate the keys. On the other hand this breaks down when git itself is running background processes, such as git-maintenance, so that’s not something that I could use here. So far out of (literally) thousands of codebases & repos I’ve used that maintenance setup exactly once, for convenience. For me it is not a major loss, then.
The one bit that feels a step backwards is that having the SSH keys in the password manager and carrying it around counteracts the “separate key for each system” arrangement. This might just be part of getting used to new processes, and not an actual downside.
Further Thoughts
In cybersecurity yesterday’s best practices might be inadecvate today and “last week’s” practices might be outright dangerous… Gonna keep revisiting this setup more broadly and in terms of details, as I learn more.
It’s a good question why even do code signing (besides having a “verified” check mark, which alone doesn’t mean much if not part of a verification process), though this needs some more space to unpack. For the time being I’ll assume that signing is better than not signing, if nothing else than as a forward looking prep for better audit processes down the line.
There’s really a question around having too many things in a single password manager: nowadays it can be the complete “royal flush” of password, TOTP, SSH key, recovery codes, passkey… and likely more bits that I might not be using yet? This does make me uneasy, and likely a scale on which usability and security will adjust over time (such as. bundling and unbundling various cybersecurity aspects).
I might also actually misunderstand various things above, if so, I’d be very keen to hear, just drop me a line!