KelsonV,
@KelsonV@lemmy.ml avatar

Hmm, a quick look at the html on the logged-out view of a post with lots of comments shows no sign of a meta robots tag, so the pages are technically crawlable. this one

But it looks like only about 25 comments out of 439 are in the html on that first page view, so probably only those will be picked up by a crawl?

  • All
  • Subscribed
  • Moderated
  • Favorites
  • lemmy@lemmy.ml
  • thenastyranch
  • magazineikmin
  • Youngstown
  • khanakhh
  • InstantRegret
  • slotface
  • mdbf
  • osvaldo12
  • rosin
  • kavyap
  • tacticalgear
  • ngwrru68w68
  • everett
  • DreamBathrooms
  • JUstTest
  • ethstaker
  • Durango
  • tester
  • cubers
  • normalnudes
  • cisconetworking
  • modclub
  • GTA5RPClips
  • anitta
  • provamag3
  • Leos
  • megavids
  • lostlight
  • All magazines