The amount of web crawlers right now which just blatantly ignore robots.txt is frustrating. It's pretty clear many of these are just scraping every single website they can to train generative text AI as well.
If nothing else, there's surely got to be some clear violation of a whole lot of CC licenses from this behaviour.
Add comment