It's all relative. To add some more context with your description,
As mentioned under load my PC (5800x3D + 3080 10gb) draws between 350 and 575 watts (depends on if I have monitors plugged into my UPS and the GPU power draw, some programs draw more than others).
Idling my PC draws about 175 watts.
In sleep my PC draws about 68-80 watts.
Like the NVIDIA GPU server example, even though it's not a lot of power, in comparison to more efficient computers doing the same task it's exorbitant.
You're right that most people with a GPU won't even be running something under load for a few hours - if they don't do rendering and they don't use AI then gaming is the only thing left that can really put a GPU to use.
So then it becomes about efficiency deciding how to optimize those tasks. If AMD can push out performance relative to NVIDIA but for 100 watts less, that's the difference between a PC in idle and a PC in sleep. That's pretty sizeable to ignore, even if you just leave the PC on 24/7 as a gaming PC+ready to use web browser. Similarly, if I'm deciding to put my GPU to use at all, it seems reasonable to consider long term cost efficiency. It's weird to think about since we don't push it much, but 20 hours a week gaming even 5 years ago vs. today is a huge power difference. Just look at the 1080Ti, a beast back then and still holds up today. Draws only 300 watts under load, and the 980ti can get 250 watts.
In terms of performance, 450w for even the 3090 let alone the 4090 absolutely blows these out of the water, but in terms of long term idle they are also, relatively, much more expensive.
All in all, most people aren't putting their PC under load 24/7 but most people also aren't only turning it on as needed. While it's true that they're not consistently drawing 300+ watts all the time, they are still likely idling (on just not being used actively) at higher levels than previous generations. My idle is quite close to the 980ti under load which is pretty insane.