We, as omnivores, have a choice. The carnivores do not. I’d rather not cause more suffering than I have to (since I have that choice) even if there was the potential that it could possibly decrease overall suffering.
I will not go into other problems with fish specifically since it’s not on-topic.