rrwo, At work we managed to contact somebody who was crawling our site with a strange bot.
The said that they were using an academic dataset of URLs that was licenced to be "used freely for any purpose".
There's a difference between using the data (a list of URLs) and actually requesting those URLs.
Add comment