Even though #ChatGPT usage just went down 10%, #AI is still a (the?) favorite tech topic. As a change to all the horror predictions, I decided to have a look at how to control indexing of web content by AI crawlers.
For traditional search engines, we have plenty of control over crawling (robots.txt etc.), preview information (OpenGraph and friends) or presentation (oEmbed).
Even though many in the end might decide to have most of their content available to #AI#crawling, I still think it is important to have controls and transparency. Especially if big companies look like they are making money with our content, without compensation.
#TDMrep by #W3C seems a good approach; and maybe also something might come out of the #Google proposal on upgrading robots.txt. But none of them will be usable in the near future.
Add comment