pussreboots, 3 months ago If you run your own website, add User-agent: ClaudeBot Disallow: / to your robots.txt This bot, run by Claude.ai on Amazon's cloud service is out of control. In 72 hours it ended up taking $2000 worth of bandwidth overage charges, In 26 of running my site I've never seen that much demand from anything. #claudebot #claudeai
If you run your own website, add
User-agent: ClaudeBot Disallow: /
to your robots.txt
This bot, run by Claude.ai on Amazon's cloud service is out of control. In 72 hours it ended up taking $2000 worth of bandwidth overage charges, In 26 of running my site I've never seen that much demand from anything.
#claudebot #claudeai
flameeyes, 3 months ago @pussreboots @stuartl I see that instead of getting rid of Semalt we now have a bunch of them? https://flameeyes.blog/2014/08/06/antibiotics-for-the-internet-or-why-blocking-semalt-crawlers/?mtm_campaign=social&mtm_kwd=mastodon https://flameeyes.blog/2010/10/20/more-crawlers-hatred/?mtm_campaign=social&mtm_kwd=mastodon
@pussreboots @stuartl I see that instead of getting rid of Semalt we now have a bunch of them?
https://flameeyes.blog/2014/08/06/antibiotics-for-the-internet-or-why-blocking-semalt-crawlers/?mtm_campaign=social&mtm_kwd=mastodon
https://flameeyes.blog/2010/10/20/more-crawlers-hatred/?mtm_campaign=social&mtm_kwd=mastodon
Add comment