Weird bug related to HDMI/DisplayPort and refresh rate and switching between TTYs.

This is what it looks like:
HDMI 60 Hz and I switch from TTY1 (with Cinnamon) to TTY2 - it takes 7 seconds.
HDMI 60 Hz return from TTY2 to TTY1 - takes 1 second.
DisplayPort 60 Hz and I switch from TTY1 (with Cinnamon) to TTY2 - it takes 6 seconds.
DisplayPort 60 Hz return from TTY2 to TTY1 - takes 6 seconds!.

On HDMI, the return takes only a second. And that's how it should always be, right?
Yet every other case lasts 6-7 seconds. That's a ridiculously long time, and there's no reason for it to take that long, is there?

In turn, at higher refresh rates, even HDMI recovery takes 6-7 seconds.
This is also surprising behavior. As if the system/drivers(?) only supported HDMI 60 Hz exactly, but not completely, because switching to TTY2 still takes too long.

What can I do with this? Is it only possible to count on driver updates to improve things?

Monitor Gigabyte M32QC Curved 31,5"
GeForce RTX 3060 Ti 8GB
ASRock A620MHDV/M.2+
AMD RYZEN 5 7500F AM5
Kingston Fury Beast Black 16GB DDR5 (KF556C40BB-16)