_elena,
@_elena@mastodon.social avatar

Geeky action du jour: adding code to all my websites to prevent from scraping their content.

I can't believe I've been putting it off for so long. FYI it's super quick and easy - you just have to add these lines to your site's robots.txt file:

User-agent: GPTBot
Disallow: /

Jgmeadows,
@Jgmeadows@mstdn.ca avatar

@_elena Does ChatGPT respect robots.txt?

_elena,
@_elena@mastodon.social avatar

@Jgmeadows ha! one would hope they do. The code comes from them (!!!) https://platform.openai.com/docs/gptbot

Jgmeadows,
@Jgmeadows@mstdn.ca avatar

@_elena I hate to be cynical. But these days it’s funny how “accidents” happen with code from large corporations and organizations…

  • All
  • Subscribed
  • Moderated
  • Favorites
  • ChatGPT
  • DreamBathrooms
  • khanakhh
  • GTA5RPClips
  • Youngstown
  • slotface
  • everett
  • tacticalgear
  • ethstaker
  • thenastyranch
  • rosin
  • kavyap
  • mdbf
  • InstantRegret
  • magazineikmin
  • bokunoheroacademia
  • Leos
  • cisconetworking
  • modclub
  • lostlight
  • osvaldo12
  • cubers
  • Durango
  • anitta
  • tester
  • relationshipadvice
  • normalnudes
  • HellsKitchen
  • sketchdaily
  • All magazines