The issue with Valheim is it’s not really a dedicated server. It’s more of a shared saved game repository.
Whoever gets to a region first is the host, and everyone nearby connects to them. If that person has a slow machine, or high latency to the other players it can be really painful. When I was playing with friends, we banned one person from entering dungeons first because they had the worst computer in the group and it was really bad for the rest of us. Same thing with our main base, if they wound up the host, they’d disconnect and log back in so someone else would become host.