Without downvotes, you end up with Twitter-tier garbage discourse and arguments because there's no way for the community to vote to remove an idiot's platform, just clap-backs and ratios.
The up/downvote system is far from perfect, but I'd really rather not lose it as the fediverse draws more people in. I would much rather have my opinion hidden on occasion because I pissed off a wasp hive than have the content I consume be filled with toxic bait that can only be hidden if a moderator gets involved.