galdor, TIL you can pass parameters to each public key in ~/.ssh/authorized_keys files. For example you can restrict connections based on the source address with from="<addr1>, <addr2>,…".
See the man page of sshd ("AUTHORIZED_KEYS FILE FORMAT") for documentation.