I get that, there's a bunch of unethical associations with everything we use. But right now the main developer of Lemmy has his profile setup with Chairman Mao as his background, his Github repo avatar is Che Guevera, and he has an entire repo filled with propaganda copy-pasta. In the past people have also pointed out that lemmy.ml and lemmygrad.ml resolve to the same IP address although this is no longer the case. Lemmygrad supports the DPRK and transitively one of the most horrific dictatorships in the world, and yet it's advertised on the "Join-Lemmy" website.
Even if the software itself is apolitical and we can see through the open-source nature of it that it is not going to be abused to further those agendas, there's no doubt that as that knowledge becomes more commonly known that it could severely impact the whole idea of a Federated Reddit alternative.
I guess the most important thing to consider is that the software, being open-sourced, can easily be forked if this ever becomes a problem.