We have a client site where some crawler bot has been repeatedly coming to the #drupal site for years and trying to access hundreds and hundreds of bad urls. All of the requests from one of their previous episodes caused #pantheon to bump the account up to the next tier, costing the organization hundreds of dollars more a month. Any ideas on how to keep an out of control crawler off of a pantheon site?
@mikemccaffrey this probably won't help in your use case, but Pantheon customers paying for their Advanced Global CDN (partially unlocked Fastly access) can block the source of "malicious" requesters at that level. In addition to the traffic limit changes, there are also changes coming to the AGCDN pricing. Hopefully those changes will made it more affordable for smaller organizations.
@mikemccaffrey yes, but not all configurations/VCL options can be done as self-service... and similar to my motivation for writing https://www.drupal.org/project/pantheon_autopilot_toolbar, you can't actually navigate to https://agcdn.ps-pantheon.com/ through the Pantheon's UI. The Edge Cache icon in the left hand navigation of the Pantheon Dashboard links to a page promoting the AGCDN and telling you to contact your sales person... even if you are paying for it.
@mikemccaffrey
There is @CrowdSec to solve this problem. And the #drupal integration at https://www.drupal.org/project/crowdsec allows you to use their shield and also automatically ban any IP for a configurable period of time, if they're crawling your site that way. E.g. after 3 consecutive 404s, banned.
@froboy Define "block". I can tell the crawler is this idiot because it has been crawling the same non-existent .html addresses for over five years now, but all I can do it return errors at it which it ignores.
@mikemccaffrey@froboy You would need to do the block at the CDN layer before it ever gets to pantheon to avoid the traffic hitting pantheon. Cloudflare has decent bot management tools you can use beyond just blocking a specific IP. My guess is it would easily identify this crawler and every script kiddie trying to reach wp-login.
Add comment