slink, 1 month ago

@rrwo unsolicited psa: always run a reverse + forward dns check to validate it's really googlebot https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot#automatic

juandesant, 1 month ago

@rrwo what about reporting as per https://about.you.com/es/youbot/?

rrwo, 1 month ago

@juandesant

Thanks, but I think there's a difference between reporting a bot that seems to be making an honest mistake due to software bugs, vs one that is outright lying about who they are when they were told not to index a site.

Edit: also, I don't want to let these sort of people know that I've caught on, lest they try and be more deceptive.

juandesant, 1 month ago

@rrwo in any case, what you report is in violation of what they say they do. Maybe it was a third party trying to be YouBot, and then GoogleBot?