It isn't "arbitrary" though. ActivityPub is just a baseline protocol that supports interoperability. Apps like Lemmy and Kbin build upon that framework, but also implement their own unique features and interfaces.
There's definitely value to being able to specifically search for Lemmy instances or things coming from Lemmy as much as any other fediverse app. But to your point, that could be handled through a filter on a much larger whatever set of data.