1️⃣3️⃣ Here's the 13th installment of posts highlighting key new features of the upcoming v256 release of systemd.
ssh is widely established as the mechanism for controlling Linux systems remotely, both interactively and with automated tools. It not only provides means for secure authentication and communication for a tty/shell, but also does this for file transfers (sftp), and IPC communication (D-Bus or Varlink).
It relies on TCP as network transport, which is great for remote operation around the globe but really sucks for local communication with a VM and similar, as it usually requires delegation of an address space, dhcp lease, dns and so on, which while manageable are certainly a major source of mistakes, fragility and headaches. In particular it means that logging into a system to debug networking doesnt really work since without working networking you cant even log in. Sad!
Thing is though, that there are better ways to communicate locally: AF_VSOCK is an alternative to AF_INET/AF_INET6 for local communication between VMs and hosts. In many ways it behaves very similar to TCP. It is similar enough so that you can just do ssh-via-AF_VSOCK. As opposed to AF_INET/AF_INEt6 it requires next to no configuration, your really just have to enable the knob in your VMM, and have a somewhat non-ancient Linux distribution as guest.
And ssh-via-AF_VSOCK is precisely what we are doing in systemd v256: a small new unit generator (i.e. a plugin that extends systemd's unit tree dynamically) detects if AF_VSOCK is available and sshd is installed, and if so binds AF_VSOCK/port 22 to sshd, via socket activation. Or in other words: in environments where AF_VSOCK is a thing, sshd will now just work, without any extra configuration and at minimal cost of resources (because lazy socket activation rocks).
Binding sshd to AF_VSOCK is one thing, but it would be useless if we couldn't also connect to it via an ssh client. Hence v256 also includes a small client-side plugin for ssh that makes it possible to connect to such a VM by specifying "ssh vsock/$CID" (where $CID is the vsock address — one of which each VM gets automatically assigned).
And there's more: the generator also binds sshd to two AF_UNIX sockets for good measure given we are already at it and it's easy. One is a socket in /run/host/ if that exists and is a mount point — which is a scheme defined for full-OS containers: the idea ist that the container manager makes that dir also available on the host, so that you can connect from the host to the container via ssh-over-AF_UNIX without bothering with network config and setup.
The other one is at a generic socket in /run/ which happens always, and can be used to acquire privs locally.
This automatic ssh-via-AF_VSOCK logic is particularly useful in conjunction with the notification mechanism detailed in installment 3, i.e. sd_notify() notifications from PID 1 to its VMM via AF_VSOCK.
Because it means a VMM can start a VM, wait for the X_SYSTEMD_UNIT_ACTIVE=ssh-access.target sd_notify() message and then immediately connect to the VM via ssh-over-AF_VSOCK. Reliably, without wasting time, without retries, without network config and setup, without preparing the guest much (well you do have to add sshd to it), just like that.
And that's all for this time. We still have more to cover, hence stay tuned.
@pid_eins Wow I've never heard of the possibility of client-side plugins for ssh! Do they work everywhere the regular ssh is used? Like from scripts, or implicitly invoked by git and the like?
@pid_eins I think I remember some distros (Arch was one of them) switching from socket activates ssh to it running as a service. IIRC the reason was because it was possible to DOS it when it was socket activated. Would this still be the case or am I missing something?
@NekkoDroid So there's a programmable rate limit on socket units. If you hit that too often we assume that something is wrong (i.e. that the service in question dies quickly after activation and never can process its connections). Hence, in order to not consume unbounded CPU we will eventually give up trying to activate the service. This safety feature can be seen as a DoS, if there simply are very high amounts of connections, the rate limit is hit too after all, it cannot…
@NekkoDroid …distinguish between very high number of attempts because of misconfiguration or because of valid connection attempts.
Now, the this ratelimit was configurable since a long long time, people just never bothered to. You can simply turn it off in the socket unit via TriggerLimitIntervalSec=/TriggerLimitBurst=.
So, the DoS never really was a DoS, it was at best a misconfiguration.
With v255 we added a separate ratelimit to socket units that behaves differently, btw.
@NekkoDroid Specifically, there's now PollLimitIntervalSec=, PollLimitBurst=. If this rate limiting is hit, then the effect is not that the socket unit will be put in a failed state, but instead we'll just stop watching the socket for the rest of the time window. Once the time passed we'll start watching again. This basically means, that if the system is hit with a flood of connection attempts, we'll pause processing them for a while, and then return processing them after the configurable…
With that in effect I think we should be really safe and robust against DoS, because we actually never deny service, but at the same time we also don't let our CPU use grow unbounded.
@pid_eins "...and that is why we are now introducing systemd-ssh. Comes with native support of xz compression, graciously contributed by Jia Tan. Enjoy!"
Add comment