I had a similar idea: Could search engines be broken up and distributed instead of being just a couple of monoliths?
Reading the HN thread, the short answer is: NO.
Still, its fun to imagine what it might look like if only…
I think the OP is looking for an answer to the problem of Google having a monopoly that gives them the power to make it impossible to be challenged. The cost to replicate their search service is just so astronomical that its basically impossible to replace them. Would the OP be satisfied if we could make cheaper components that all fit together to make a competing but decentralized search service? Breaking down the technical problems is just the first step, the basic concepts for me are:
Crawling -> Indexing -> Storing/host index -> Ranking
All of them are expensive because the internet is massive! If each of these were isolated but still interoperable then we get some interesting possibilities: Basically you could have many smaller specialized companies that can focus on better ranking algorithms for example.
What if crawling was done by the owners of each website and then submitted to an index database of their choice? This flips the model around so things like robots.txt might become less relevant. Bad actors and spam however now don’t need any SEO tricks to flood a database or mislead as to their actual content, they can just submit whatever they like!. These concerns feed into the next step:
What if there were standard indexing functions similar to how you have many standard hash functions. How a site is indexed plays an important role in how ranking will work (or not) later. You could have a handful of popular general purpose index algorithms that most sites would produce and then submit (e.g. keywords, images, podcasts, etc.) combined with many more domain specific indexing algorithms (e.g. product listings, travel data, mapping, research). Also if the functions were open standards then it would be possible for a browser to run the index function on the current page and compare the result to the submitted index listing. It could warn users that the page they are viewing is probably either spam or misconfigured in some way to make the index not match what was submitted.
What if the stored indexes were hosted in a distributed way similar to DNS? Sharing the database would lower individual costs. Companies with bigger budgets could replicate the database to provide their users with a faster service. Companies with fewer resources would be able to use the publicly available indexes yet still be competitive.
Enabling more competition between different ranking methods will hopefully reduce the effectiveness of SEO gaming (or maybe make it worse as the same content is repackaged for each and every index/rank combination). Ranking could happen locally (although this would probably not be efficient at all but that fact that it might even be possible at all is quite a novel thought)
The only way I can imagine this working is by twisting the definition of the words "search engine" enough that you can claim that there aren't search engines, but really there are still, just under a different name.
Search engines aren't actually the "problem" that OP is wanting to address, here, though. He just doesn't like the specific search engines that actually exist right now. What he should really be asking is how a search engine could be implemented that doesn't have the particular flaws that he's bothered by.
Or AI to rank and filter out the things you need based on public indexing. Preferably there’d be several AI assistants to choose from. Things seem to be moving in that direction anyway.
While this is true (and a problem with current engines like Google), I could see having a local LLM doing the filtering for you based on your own criteria. Then you could do a wide-open search as needed, or with minimal filtering, etc.
When I’m searching for technical stuff (Android rom, Linux commands/how it works), it would be really helpful to have some really capable filtering mechanisms that have learned.
When I want to find something from a headline, then it needs to be mostly open (well, maybe filtering out The Weekly World News).
But it really needs to be done by my own instance of an LLM/AI, not something controlled elsewhere.
Given that the indices are not available locally, it’d be difficult for your own algorithm of any sort, AI or otherwise, to rank items higher/lower than others.
That’s pretty much what all of the site aggregators were. I ran a couple of communities on yahoo and some other sites. There were also services like Archie, gopher, and wais, and I am pretty sure my Usenet client had some searching on it (it might have been emacs - I can’t remember anymore). I remember when Google debuted on Stanford.edu/google and realized that everything was about to change.
Pay to play was the problem there. I had the highest ranking joke page on webcrawler for a stint, but Yahoo wanted $500 to put me on top. My 15 year old self was not interested.
I have a very difficult time imagining an internet that is both interoperable and ranking-free. Now, that having been said, we are well outside my area of expertise here so I’d love to hear from folks who know more than me.
What about just giving transparency to what the ranking is and letting people control it? Analogous to “sort by new/best/top” bit ideally with more knobs to tweak and a bunch of preset options?
Add comment