For this I used the information from the blog below, which misses some of yours, but has some others. And some other suggestions too, such as going beyond robots.txt and using the newly minted ai.txt.
@cory another great piece! I especially like what you said about Copilot, "a mild improvement over traditional autocomplete" which is an interesting comparison. We've had autocomplete for so long, but it ALWAYS needs to be checked, just like Copilot. I never/very rarely even use autocomplete. I have to edit and reread all my own writing anyway, WHY would I add another layer?! 😂
@jake4480 Thanks bud! And that’s with me using just the LSP in Sublime, not the whole VS Cose integration (it really bothers me how much of the dev tool chain MSFT has scooped up). ChatGPT is even worse and totally untrustworthy imho.
@cory it's insane. And like in another piece I saw, it feels like EVERY DAY we're seeing lists of companies adding it. I hope/feel like it's a bubble/all hype. Like many do. My question is, do companies honestly think AI is the future, or are they begrudgingly adding it in, like well, I guess we gotta add in this shit now to keep up with all the other companies.... 🤣
@cory absolutely. Investors. It's gonna mess stuff up and/or potentially evolve into something even more stupid. I can't imagine it. But if there's anything I've learned, is that there's ALWAYS something more stupid right around the bend. 🤣
@cory oh man. You put them all in one sentence like that, and wow. No. Yuck. 🤣 Here is great if it counts as social media, I guess it does. But all the big crappy companies that do it, nah. It's questionable benefit over even such a short amount of time. Sometimes I think about that. The thousands of years of humanity (and even LESS with tech) in the scheme of the millions of this planet, it's wild. 🤣 such a tiny blip.
@jake4480 Hahaha — we never should’ve privatized the public discourse (and so many other things but 🇺🇸). I saw someone else make a comment the other day along the lines of 200 years ago the average lifespan was 39 years old and now look at what LeBron James is doing at that age and I keep spinning that around in my head. 😅
@cory the list has grown in the past few months! I wonder if there’s a repo or even a package that can help keep this mushrooming madness under control.
@cory any thoughts on Common Crawl bot? I understand it’s been used for AI training but if I remember correctly, it’s also used by indie search engines 🤔
@analog_cafe I’ve got it blocked myself — I respect that indie search engines use it but if it’s also leveraged by a significant number of LLMs I’d come down on the side of blocking it.
@cliophate@mako That’s very much a concern — and makes their behavior and claims to any sort of legitimacy all the more dubious. I’d imagine you could redirect crawlers based on user agents at the server/request level as well, but that assumes they’re setting anything in good faith.
@sangster Ah yep! I’ve got mine in a static array that I update and expose via a data file and template that loops through that to spit the find file out.
@sangster I just fixed mine recently! The date format was wrong and the final output had p tags in it because it was being generated from a markdown rather than liquid file 🤦🏼♂️
Add comment