Usually a reverse proxy runs behind the firewall/router. The idea you are pointing 80/443 at the proxy with port forwarding once traffic hits your router.
So if someone goes to service.domain.com
You would have dynamic DNS telling domain.com the router is the IP.
You would tell domain.com that service.domain.com exists as a cname or a record. You could also say *.domain.com is a cname. That would point any hosttname to your router.
From here in the proxy you would say service.domain.com points to your services IP and port. Usually that is would be on the lan but in your case it would be through a tunnel.
It is possible and probably more resource efficient to just put the proxy on the VPS and point your public domain traffic directly at the VPS IP.
So you could say on the domain service.domain.com points to the VPS IP as an a record. Service2.domain.com points to the VPS IP as another a record.
You would allow 80/443 on the VPS and create entries for the services
Those would look like the service.domain.com pointing to localhost:port
In your particular case I would just run the proxy on the public VPS the services are already on.
Don’t forget you can enable https certificates when you have them running. You can secure the management interface on its own service3.domain.com with the proxy if you need to.
And op consider some blocklists for your vps firewall like spamhaus. It wouldn’t hurt to setup fail2ban either.