cory, 1 month ago Added some more #AI robots to block over on GitHub https://github.com/ai-robots-txt/ai.robots.txt #WebDev
Added some more #AI robots to block over on GitHub https://github.com/ai-robots-txt/ai.robots.txt #WebDev
bartosz, 1 month ago @cory Rage Against the Machine Learning
@cory Rage Against the Machine Learning
croc, 1 month ago @cory Ya know, a robots.txt doesn't actually prevent bots from scraping your site, right? It is basically just requesting the bot to not do it, but it is within its means and rights to ignore it. Yes, it is unethical though.
@cory Ya know, a robots.txt doesn't actually prevent bots from scraping your site, right? It is basically just requesting the bot to not do it, but it is within its means and rights to ignore it. Yes, it is unethical though.
Add comment