Every server has it's own instance of Nginx Proxy Manager (not needed, but helps with using docker networking), then every service I run on docker, I just use docker networks to talk to the rproxy.
My network is designed such that all my externally available services are on one server, so I just forward 80 & 443 to that.
For every other service (not on docker), I just use 127.0.0.1 + port.
For services on other servers on my LAN, I use Internal IP + port.