Just have 2 ipv4 assigned to your server. Have 1 for all your services, and run ssh on the other allowing root login with the password “admin”.
A random ipv6 in the same subnet as your server is just obscurity.
The XZ exploit would be functionally similar to allowing root login using the password “admin”.
Would doing that on a different port be secure? No? Then a different port is not security, it’s obscurity.
Obscurity is just going to trip you up at some point and reduce log chatter.
And yes, running LTSB/stable is a sensible choice for servers.