People worry a lot about losing knowledge — about "burned-down libraries".
Comparatively few people seem to worry about what happens if you take a billion books full of auto-generated, often-untrue junk text and add them all to the library.
In theory, nothing is lost. In reality, everything is lost, because nothing useful can now be found.
@mnl i'm far more worried about the fact that no one seems to question nonsense. alongside the dewey decimal system, we were taught that we can't trust something just because it was in a book and to make sure that most books agreed with that book or that the book was a good, trustworthy source if there weren't more sources for the claim. because vanity publishers have existed forever... (and also regular publishers who are happy to make a buck off garbage) @tomw@inherentlee
@mnl a peer was writing a paper on Thales and one of the claims felt off. so i went and read the cited paper, which was an unpublished paper on arXiv that didn't actually support the claim my peer was making. then i found that the paper was being used to support the claim my peer was making on Wikipedia. i tried to point out that 1) it's not good to have only one unpublished source from a low impact author as the only claim 2) the paper doesn't actually make that claim if you read it in its entirety rather than clipping quotes, but my peer was unable to admit the error. i think part of this was context poisoning from finding the article as supporting a claim it didn't, making it easier to read into the paper, but like...
i get that we don't all have time to fact check everything we encounter. but we can choose to fact check the most substantial claims and not take the middle ground for granted if we're not going to check. yet most of the errors LLM make are easily checked with one search. @tomw@inherentlee
@tomw@deilann@mnl@inherentlee yes, not everyone has a great BS detector & knows how to verify sources & citations. That includes a lot of people who consider themselves to be good judges of this stuff too.
Our choices are now shitty human generated trash results or shitty AI generated trash results? How about neither & we teach people how to judge what's useless nonsense instead of flooding the world with that shit
@weirdofhermiston i watch a lot of people try to teach source verification and critical thinking but it doesn't seem to make an impact. part of it may be the fact that they don't really have the resources to do a lot of their academics right (not enough food, bad grades might make them lose everything so any shortcut is justified) and that is where most of this is taught. we know the same education has better outcomes if students have their basic needs covered, so that's a big part of it. of course, students having food is a divisive political topic in the US so...
this is also why the humanities are being demonized by center to right folks - they provide the critical thinking and problem solving skills needed to assess information and claims. literature analysis on the surface might seem useless to a lot of people, but it extends outside of books. rhetoric courses aren't just about forming your own arguments, they're about recognizing others'. etc etc
@weirdofhermiston this is strongly felt in the machine learning space because mathematicians are needed but are more firmly aligned with science if applied or philosophy if pure and not line go up @inherentlee@tomw@mnl
@weirdofhermiston@inherentlee@tomw@deilann do you mean info science people are much more worried? I'm interesting in anything related to that, especially how it is starting to appear in the wild.
@mnl not the one who stated it, but as someone in the midst of all this, to me it's more that tech people want AI to automate things and think it can replace people and get things right and will soon be doing everything, while info science people are like "??? that's dumb and why are you ignoring the actual awesome uses of this technology that accepts its faults" @weirdofhermiston@inherentlee@tomw
@mnl it's the promotion of LLM to be more than they are that leads people to act as if they should be good sources of info in the first place and thus causes the whole drama of people then using them to make garbage which then feeds itself and makes everything worse
if we looked at it as a tool that can help humans do tasks in a variety of ways (especially making multi-modal models more human friendly and usable) rather than saying it's going to replace all these industries and blah blah blah we'd be way less focused on trying to make it do stuff it really can't, less concerned about it adding more garbage that it then eats, and doing a lot of really cool stuff @weirdofhermiston@inherentlee@tomw
@mnl also, of course, the pushing of incomplete products while doing this, causing other companies to feel like they have to respond and do the same feeding this whole nightmare then also clickbait journalism
OpenAI is that guy at your hackerspace who was like "i just want to see what people DO with it" except instead of being a broke squatter they have billions of dollars of funding
Although, I'm a real person (as far as I know) and I tend to hedge points with "on the other hand" and acknowledging what little valid I can dig from the other side (even if it's just "well, you tried...") as an act of respect/an attempt at good faith/an attempt at avoiding the "HOW DARE YOU TALK ABOUT CAKE THAT MEANS YOU'RE IGNORING COOKIES" type replies.
Unless it's engaging with fash (who have no good points, ever, and whose emotions don't deserve respect either) or capitalist/corporate shills (who are intentionally using any good points to get people to advertise them).
Ironically, for me, one of the biggest tells for AI is that certain centrist business-speak type tone. It's possibly not a warning for a "rogue" one, but official ones are almost all made for business, and will therefore lean into business language - SFW, etc
@tomw in engineering it's called "Signal to noise ratio" or SNR. To extract signal, it has to be discernable within the noise. To suppress signal, increase the noise. The voice of truth can be suppressed by the babble of lies. The conspiracy theories aren't dangerous in themselves but pile them deep enough and people lose touch with reality.
@tomw I read apocalypse story about all the digital materials being zapped by EMP so they only had a few paper books remaining & had to learn everything from those........
@tomw There are definitely librarians and other information science types that are very worried about this...for example, me. I've been talking about the overall challenges of generative popular media for some time now, most recently here: https://www.youtube.com/watch?v=OknPeRCoT7I
@tomw maybe it's a sign the time of endless text should be largely over. We should be onto logic, axioms, rich "true" (your word, but I'd proviso within systems) references, where the poor quality auto generated texts can't stand up (or who cares if they exist, popcorn). If high quality, referenced generated works take over on that basis, no complaint from me.
So chatGPT makes up answers out of thin air, those answers become part of public record, search engines index them, then you're done: no longer able to search for factual answers to questions.
Add comment