alexr, 16 days ago I'd like a change to robots.txt that instructs not-mentioned robots to visit a special page to accept terms and conditions for scraping the site.
I'd like a change to robots.txt that instructs not-mentioned robots to visit a special page to accept terms and conditions for scraping the site.