I just left it on, I don't really understand it though. I still fail even trivial skill checks repeatedly with anyone other than Astarion so I'm not actually convinced it does anything at all.
I know there is a kind of philosophy in random dice roll design for software where you slightly skew the distribution of subsequent rolls based on previous ones because it feels more random (rather than getting e.g. 1, 2, 1, 2, which could be 'truly random' but feels like ass), but I don't think that's what karmic dice is doing. I do play really high AC, with Laezel and Shadowheart in full endgame plate plus a buff here or there I think they run 24~27. It is very hard for enemies to hit us and it feels about right. I would be very surprised to learn that karmic dice is benefiting enemies too, because that seems unintended and not in the spirit of how the tooltip is written.