Someone could literally make a website listing downvotes throughout the fediverse, and there's nothing stopping them.
This is why I agree that it should be shown upfront. A lot of people won’t like it, but I think users should be somewhat aware that it’s all technically visible.
Someone is gonna make an instance that does exactly this at some point. It will be inevitable as the fediverse matures.