I think the solution is to get hardware-accelerated transcoding set up correctly.
Either that, or maybe bulk-transcode all your media outside of Jellyfin and then add it all back as alternate versions and hope Jellyfin is smart enough to choose the best supported version automatically?
(I’m in the process of setting up a Jellyfin server and noticing the same issue with some of my media.)