underlap, 1 month ago (edited 1 month ago) Here's the beginnings of a small project to create a NGINX module to enforce the rules in robots.txt, for AI web crawlers that ignore the rules. New Post: NGINX robot access, https://underlap.org/nginx-robot-access. #rustlang #nginx #robots
Here's the beginnings of a small project to create a NGINX module to enforce the rules in robots.txt, for AI web crawlers that ignore the rules.
New Post: NGINX robot access, https://underlap.org/nginx-robot-access.
#rustlang #nginx #robots