mikemccaffrey, 3 months ago @froboy Define "block". I can tell the crawler is this idiot because it has been crawling the same non-existent .html addresses for over five years now, but all I can do it return errors at it which it ignores.
@froboy Define "block". I can tell the crawler is this idiot because it has been crawling the same non-existent .html addresses for over five years now, but all I can do it return errors at it which it ignores.