Sorry man, I don’t know what to tell you. I’ve got a pretty medium end VPS on which I host my Matrix instance - only had to add an extension for storage after the first few years when the DB got too big. Things were never as bad as you said early on, and as time passed I absolutely got to the point where it would take 10-20 seconds to sync - but this was after 2 years or so of constant use.
The reason why it takes long is because of the size of the sync payload - logically, for a new server/user, this really shouldn’t be that big (unless you’re in rooms like Matrix HQ). So, genuinely, look into optimization: postgres, your web server (nginx, apache, caddy), and limiting your users from accessing “problematic” rooms.
Barring that just deploy the sliding sync proxy and be done with it. It’s not really a problem that requires you to attempt it a thousand times.
So either you put some fancy wizardry into your system or you’re just in denial.
It’s called pure Debian, baby. Also, you’ll need a decent chunk of RAM if you don’t have that yet. Avoid a pagefile if you can.