nixCraft, 21 days ago In a deliciously ironic twist, OpenAI's website forbids scraping... lol.
In a deliciously ironic twist, OpenAI's website forbids scraping... lol.
nixCraft, 21 days ago FYI, you can block OpenAI, Google AI and others with robots.txt now https://www.cyberciti.biz/web-developer/block-openai-bard-bing-ai-crawler-bots-using-robots-txt-file/
FYI, you can block OpenAI, Google AI and others with robots.txt now https://www.cyberciti.biz/web-developer/block-openai-bard-bing-ai-crawler-bots-using-robots-txt-file/
fj, 21 days ago @nixCraft I wish there was the possibility to specify the usage of the scraping rather than the agent. https://mastodon.social/@fj/112280775190792281
@nixCraft I wish there was the possibility to specify the usage of the scraping rather than the agent.
https://mastodon.social/@fj/112280775190792281
dozymoe, 21 days ago @nixCraft how about X's AI?
@nixCraft how about X's AI?
ErikJonker, 21 days ago @nixCraft ..but good to remember it won't help against scrapers, crawlers who just ignore robots.txt , i don't think a chinese crawler bot will care for example.
@nixCraft ..but good to remember it won't help against scrapers, crawlers who just ignore robots.txt , i don't think a chinese crawler bot will care for example.
dozymoe, 21 days ago @nixCraft they don't want to accidentally incest other AI scraper
@nixCraft they don't want to accidentally incest other AI scraper
nixCraft, 21 days ago this was original but they recently updated it
this was original but they recently updated it
JeanFrancoisKennedy, 21 days ago @nixCraft doesn't this remove search engine referencing all together?
@nixCraft doesn't this remove search engine referencing all together?
Add comment