I’m curious why instances offering free search are defederated? I would have guessed everyone wants better search. Is it because of privacy concerns or instances don’t want to be indexed or have traffic directed elsewhere?
I was hoping that if I index only for the purpose of embeddings (which would prevent recreating the original content) and only share urls to the content that it should eliminate privacy and traffic concerns.
I’m still in the process of understanding how and if this would work. It’s only a personal project at this stage but you are right cpu/gpu and vector stores would be things I’d need to consider.