Seeking assistance setting up traefik with wireguard server

I’m hoping someone can help me figure out what I’m doing wrong.

I have a VM on my local network that has Traefik, 2 apps (whomai and myapp), and wireguard in server mode (let’s call this VM “server”). I have another VM on the same network with Traefik and wireguard in client mode (let’s call this VM “client”).

  • both VMs can can ping each other using their VPN IP addresses
  • wireguard successfully handshakes
  • I have myapp.mydomain.com as a host override on my router so every computer in my house points it to "client"
  • when I run curl -L --header ‘Host: myapp.mydomain.com’ from the myapp container it successfully returns the myapp page.

But when I browse to http://myapp.mydomain.com I get “Internal Server Error”, yet nothing appears in the docker logs for any app (neither traefik container, neither wireguard container, nor the myapp container).

Any suggestions/assistance would be appreciated!

shasta,

500 errors typically log a stack trace in the server logs. Have you checked there? That would give more indication of where to start debugging.

mike_wooskey,
@mike_wooskey@lemmy.d.thewooskeys.com avatar

By “server log”, do you mean traefik’s log? If so, this is the only thing I could find (and I don’t know what it means): lemmy.d.thewooskeys.com/comment/514711

shasta,

No. Traefik says the 500 error came from downstream. So that means either wireguard or myapp. Check the logs for those.

deergon,
@deergon@lemmy.world avatar

Just a few thoughts:

  • Did you enable access logs in Traefik as well as setting global log level to debug? This usually gives a lot more info about whats going on
  • Are the containers using the same docker network or host network, so they can reach each other?
mike_wooskey,
@mike_wooskey@lemmy.d.thewooskeys.com avatar

Thanks for helping, @deergon.

Both traefik containers (on the “server” and “client” VMs) and the wireguard server container were built with TRAEFIK_NETWORK_MODE=host. The VMs can ping each other and the Wireguard containers can ping each other.

Both traefik containers were built with TRAEFIK_LOG_LEVEL=warn but I changed them both to TRAEFIK_LOG_LEVEL=info just now. There’s a tad more info in the logs, but nothing that seems pertinent.

deergon,
@deergon@lemmy.world avatar

How about the Traefik access logs (separate from the main log), do they reveal anything?

mike_wooskey,
@mike_wooskey@lemmy.d.thewooskeys.com avatar

From traefik’s access.log:


<span style="color:#323232;">{"ClientAddr":"192.168.1.17:45930","ClientHost":"192.168.1.17","ClientPort":"45930","ClientUsername":"-","DownstreamContentSize":21,"DownstreamStatus":500,"Duration":13526669,"OriginContentSize":21,"OriginDuration":13462593,"OriginStatus":500,"Overhead":64076,"RequestAddr":"whoami.mydomain.com","RequestContentSize":0,"RequestCount":16032,"RequestHost":"whoami.mydomain.com","RequestMethod":"GET","RequestPath":"/","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"websecure-whoami-vpn@file","ServiceAddr":"10.13.16.1","ServiceName":"whoami-vpn@file","ServiceURL":{"Scheme":"https","Opaque":"","User":null,"Host":"10.13.16.1","Path":"","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"StartLocal":"2024-04-30T00:21:51.533176765Z","StartUTC":"2024-04-30T00:21:51.533176765Z","TLSCipher":"TLS_CHACHA20_POLY1305_SHA256","TLSVersion":"1.3","entryPointName":"websecure","level":"info","msg":"","time":"2024-04-30T00:21:51Z"}
</span><span style="color:#323232;">{"ClientAddr":"192.168.1.17:45930","ClientHost":"192.168.1.17","ClientPort":"45930","ClientUsername":"-","DownstreamContentSize":21,"DownstreamStatus":500,"Duration":13754666,"OriginContentSize":21,"OriginDuration":13696179,"OriginStatus":500,"Overhead":58487,"RequestAddr":"whoami.mydomain.com","RequestContentSize":0,"RequestCount":16033,"RequestHost":"whoami.mydomain.com","RequestMethod":"GET","RequestPath":"/favicon.ico","RequestPort":"-","RequestProtocol":"HTTP/2.0","RequestScheme":"https","RetryAttempts":0,"RouterName":"websecure-whoami-vpn@file","ServiceAddr":"10.13.16.1","ServiceName":"whoami-vpn@file","ServiceURL":{"Scheme":"https","Opaque":"","User":null,"Host":"10.13.16.1","Path":"","RawPath":"","OmitHost":false,"ForceQuery":false,"RawQuery":"","Fragment":"","RawFragment":""},"StartLocal":"2024-04-30T00:21:51.74274202Z","StartUTC":"2024-04-30T00:21:51.74274202Z","TLSCipher":"TLS_CHACHA20_POLY1305_SHA256","TLSVersion":"1.3","entryPointName":"websecure","level":"info","msg":"","time":"2024-04-30T00:21:51Z"}
</span>

All I can tell from this is that there is a DownstreatStatus of 500. I don’t know what that means.

deergon,
@deergon@lemmy.world avatar

Have you tried accessing your service url from inside the Traefik container? Eg. wget 10.13.16.1? Also you seem to be accessing the service url with https, which usually requires insecureSkipVerify=true. Otherwise you might get http-500 error downstream.

lemmyvore,

You’ll have to give more details. Where are you browsing from? How is the tunnel between the VMs relevant? Are the VMs’ IPs routed on the LAN? Is myapp.mydomain.com defined in a DNS server, and if so which? Is it the DNS server on the LAN or a public DNS? Do both VM and the machine you’re browsing from resolve that address to the same IP, and is that IP reachable from the browser machine?

mike_wooskey,
@mike_wooskey@lemmy.d.thewooskeys.com avatar

Thanks for helping, @lemmyvore.

I’m browsing from my laptop on the same network as promox: 192.168.1.0/24

The tunnel is relevant in that my ultimate goal will be to have “client” in the cloud so I can access my apps from the world while having all traffic into my house be through a VPN.

The VM’s IPs are 192.168.1.50 (“server”) and 192.168.1.51 (“client”). They can see everything on their subnet and everything on their subnet can see them.

Everything is using my router for DNS, and my router points myapp.mydomain.com and whoami.mydomain.com to “client”. And by “everything” I mean all computers on the subnet and all containers in this project.

Both VMs and my laptop resolve myapp.mydomain.com and whoami.mydomain.com to 192.168.1.51, which is “client”, and can ping it.

lemmyvore,

Is the browser also using the LAN router for DNS? Some browsers are set to use DoT or DoH for DNS, which would mean they’d bypass your router DNS.

Do you also get “Internal Server Error” if you make the request with curl on the CLI on the laptop?

How did you check that mydomain is being resolved correctly on the laptop?

What do you get with curl from the other VM, or from the router, or from the host machine of the VM?

mike_wooskey,
@mike_wooskey@lemmy.d.thewooskeys.com avatar

Thanks so much for helping me troubleshoot this, @lemmyvore!

Is the browser also using the LAN router for DNS? Some browsers are set to use DoT or DoH for DNS, which would mean they’d bypass your router DNS.

My browser was using DoH, but I turned it off and still have the same issue.

Do you also get “Internal Server Error” if you make the request with curl on the CLI on the laptop?

Yes, running curl -L -k --header ‘Host: whoami.mydomain.com’ 192.168.1.51 on the laptop results in “Internal Server Error”.

How did you check that mydomain is being resolved correctly on the laptop?

ping whoami.mydomain.com hits 192.168.1.51.

What do you get with curl from the other VM, or from the router, or from the host machine of the VM?

From the router:


<span style="color:#323232;">Shell Output - curl -L -k --header 'Host: whoami.mydomain.com' 192.168.1.51
</span><span style="color:#323232;">  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
</span><span style="color:#323232;">                                 Dload  Upload   Total   Spent    Left  Speed
</span><span style="color:#323232;">
</span><span style="color:#323232;">  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0-
</span><span style="color:#323232;">100    17  100    17    0     0   8200      0 --:--:-- --:--:-- --:--:-- 17000
</span><span style="color:#323232;">
</span><span style="color:#323232;">100    21  100    21    0     0    649      0 --:--:-- --:--:-- --:--:--   649
</span><span style="color:#323232;">Internal Server Error
</span>

From the wireguard client container on the “client” VM:


<span style="color:#323232;">curl -L -k --header 'Host: whoami.mydomain.com' 192.168.1.51
</span><span style="color:#323232;">Internal Server Error
</span>

From the traefik container on the “client” VM:


<span style="color:#323232;">$ curl -L -k --header 'Host: whoami.mydomain.com' 192.168.1.51
</span><span style="color:#323232;">Internal Server Error
</span>

From the “client” VM itself:


<span style="color:#323232;"># curl -L -k --header 'Host: whoami.mydomain.com' 192.168.1.51
</span><span style="color:#323232;">Internal Server Error
</span>

From the wireguard container on the “server” VM:


<span style="color:#323232;"># curl -L -k --header 'Host: whoami.mydomain.com' 192.168.1.51
</span><span style="color:#323232;">Internal Server Error
</span>

From the traefik container on the “server” VM (This is interesting. Why can’t I ping from this traefik installation but a can from the other? But even though it won’t ping, it did resolve to the correct IP):


<span style="color:#323232;">$ ping whoami.mydomain.com
</span><span style="color:#323232;">PING whoami.mydomain.com (192.168.1.51): 56 data bytes
</span><span style="color:#323232;">ping: permission denied (are you root?)
</span>

From the “server” VM itself:


<span style="color:#323232;"># curl -L -k --header 'Host: whoami.mydomain.com' 192.168.1.51
</span><span style="color:#323232;">Internal Server Error
</span>
mike_wooskey, (edited )
@mike_wooskey@lemmy.d.thewooskeys.com avatar

Also, just to make sure the app is indeed running, I curled it from it’s own container (I’m using myapp here instead of whoami, because whoami doesn’t have a shell):


<span style="color:#323232;">$ curl -L -k --header 'Host: myapp.mydomain.com localhost:8080
</span>

I can’t seem to display html tags in this comment, but the results are the html tags for the web page for the app - so the app is up and running

Lem453,

This seems like an issue where the wireguard is not using the correct DNS server. Does the wireguard DNS setting point to the router?

A diagrams might help me to see what is going on more clearly.

mike_wooskey, (edited )
@mike_wooskey@lemmy.d.thewooskeys.com avatar

Thanks for helping, @Lem453.

Both wireguard containers are using my router for DNS, and my router points myapp.mydomain.com and whoami.mydomain.com to “client”.

https://lemmy.d.thewooskeys.com/pictrs/image/b62d7314-644c-4fd1-bfac-1cabb7d99292.png

mike_wooskey,
@mike_wooskey@lemmy.d.thewooskeys.com avatar

I should add that I’m running Traefik 2.11.2 and wireguard from the Linuxserver image lscr.io/linuxserver/wireguard version v1.0.20210914-ls22.

mike_wooskey,
@mike_wooskey@lemmy.d.thewooskeys.com avatar

@deergon, @shasta, and @lemmyvore,

THanks for your help. My main issue ended up being that I was trying to use Let’s Encrypt’s staging mode, but since staging certs are self-signed, Traefik was not accepting the requests. Also, though I had to switch Traefik’s logging level to Info instead of error to see that.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • selfhosted@lemmy.world
  • InstantRegret
  • ngwrru68w68
  • everett
  • Durango
  • osvaldo12
  • rosin
  • mdbf
  • DreamBathrooms
  • thenastyranch
  • magazineikmin
  • Youngstown
  • GTA5RPClips
  • slotface
  • kavyap
  • JUstTest
  • modclub
  • cubers
  • ethstaker
  • tacticalgear
  • khanakhh
  • Leos
  • cisconetworking
  • anitta
  • normalnudes
  • tester
  • megavids
  • provamag3
  • lostlight
  • All magazines
    •