The smaller community is really a blessing, rather than a curse. I've seen this come up on reddit again and again: the best subs are the small ones that cater to niche interests. Lemmy is essentially composed entirely of those small subreddits right now.