tdp_org, to SEO
@tdp_org@mastodon.social avatar

We recently noticed a fair bit of traffic on www.bbc.co.uk & www.bbc.com from a User Agent which identifies itself as "ByteSpider" (& has a @bytedance.com email address).

Lots of docs on the web state it doesn't obey robots.txt but ByteDance have told us it does:

> ...in the robots.txt files
> user-agent:Bytespider
> Disallow:/

Thought that might be worth documenting as it might be a recent change & several of us searched but found zero docs from ByteDance

  • All
  • Subscribed
  • Moderated
  • Favorites
  • JUstTest
  • everett
  • magazineikmin
  • mdbf
  • thenastyranch
  • khanakhh
  • rosin
  • Youngstown
  • ethstaker
  • slotface
  • modclub
  • kavyap
  • DreamBathrooms
  • Durango
  • provamag3
  • ngwrru68w68
  • InstantRegret
  • tacticalgear
  • GTA5RPClips
  • cubers
  • normalnudes
  • osvaldo12
  • tester
  • anitta
  • cisconetworking
  • megavids
  • Leos
  • lostlight
  • All magazines