janl, 10 months ago

@brucelawson you could make it as “this has all been LLM generated” and they will leave you alone.

LDJ, 10 months ago

@janl @brucelawson There was a discussion on this recently and unfortunately they don’t seem to be honouring the robots.txt - one site was having to individually block the domains, but there’s another one every day 😕

garrettc, 10 months ago

@brucelawson This is in my robots.txt: https://polytechnic.co.uk/robots.txt taken from https://platform.openai.com/docs/plugins/bot

ppatel, 10 months ago

@garrettc @brucelawson How do you deal with Google though?

garrettc, 10 months ago

@ppatel How do you mean? Do you want to block Google's crawler?

ppatel, 10 months ago

@garrettc Considering Google's latest privacy changes appear to have declared that the company is going to use pretty much everything on the web for training its models, I'm concerned about Google no longer honoring robots.txt.

garrettc, 10 months ago

@ppatel I think they'd be creating a lot of trouble with the web community if they decided not to honour the robots.txt standard.

ppatel, 10 months ago

@garrettc I'm not seeing the current management caring about that much these days.

patrick_h_lauke, 10 months ago

@brucelawson even if you could, would they honour it? "i'm afraid i can't do that, bruce"