Its not too bad, you can define the UID that your podmab process runs on since it can run as its own process and not be reliant on a daemon. It makes you do a little more admin work, but honestly you have to start doing that anyway in a lot of environments.
I have had good experimental experiences with podman, but my biggest barrier to adopting it is how old the packaged version with debian is. I'm hoping to see if I can do some damage with the release of bookworm.