This month's editorial looks at the possible ethics issues that could come with introducing a reliable detector to the filtering process of a submissions system. Curious what people think. https://clarkesworldmagazine.com/clarke_05_24/
What it boils down to is the potential common good (helping deprioritize potential spam in submission queues, an effective pressure valve that can keep markets open) a fair trade or is the fact that a partner used other's generated content to build their detector crossing a line.
In this scenario, the people building the detector aren't directly involved in theft from authors. It's trained on the text generated by systems involved in theft. Basically, the children of theft.
@clarkesworld Great read. Do I understand correctly that by "It's trained on the text generated by systems involved in theft," you means the text is generated by ChatGPT or some tool like it?
@grammargirl@clarkesworld Yes, and. You have to train the detector system on a bunch of generated content from ChatGPT and also content from every other LLM on the market - ethical and unethical alike. Each LLM has its own flavor. And also, the detector itself, the result of that extra training is bolted onto the base LLM like ChatGPT in a process called fine-tuning. There are lots of processes developers can work from but they usually involve both of those steps.
I don't see an ethical issue because
1: AI content is not copyrightable
B: turnabout is fair play, and
Finally: boo hoo
However, I've read several reports that show detectors for AI content to have very high false positive and false negative ratings, and are particularly likely to report the writings of real humans who have autism as AI generated, sooo "reliable" is the hardest part of your statement. I'll still be interested in reading the editorial sometime though.
@lord_tacitus Yes, and ESL too. I've been talking with developers and evaluating some new tools that (based on my own testing) are much more reliable, even in these situations. Human final review is going to be necessary for the foreseeable future, but these detectors look promising and could significantly improve the things we already do to manage this type of spam.
@clarkesworld A few thoughts: You are dealing with a special kind of spam here, call it slop spam. To catch a spammer, remember they are lazy people. They don't fake meta data well, or don't even bother.
Look at repetitive patterns in IP addresses or user agents of the submitting Browser.
don't accept submissions from outdated software, like a ten year old firefox.
don't accept submissions by people that don't bother to hide they are using curl or wget. #spam#slop#ai 1/3
@clarkesworld 2. Validate the author's email address, and check if it exists and is answerable. There are online services dealing with this. Don't accept submissions using disposable email addresses. Consider rejecting submissions from freemail services or services with a bad security record. Yes, this includes yahoo for example. #spam#slop#ai 2/3
@clarkesworld 3. The author's postal address should exist. Validation is more expensive than email address validation, but surprisingly accurate in developed countries. The same goes for the phone number. Look for patterns.
4. Give captchas a try on your submissions page, if you haven't already. #spam#slop#ai 3/3
@jotho We're already doing much of that, but validation services are not cheap, can have serious regional limitations, and (with email) would exclude a large number of legitimate authors, including many we've worked with. The email addresses are 99.9% real. The rest are typos. Keep in mind that our spammers are hoping to get paid, so they want to be reachable. Many don't even know they are being told to do something wrong. (A lot is driven by side hustle scams.)
@clarkesworld Interesting approach but I'm worried it's an arms race the little guys are bound to loose. The detector will inevitably need substantially more resources (vram = $) to maintain accuracy relative to the original LLM tokenization size. Granted, the more unique outputs need more diverse training (https://arxiv.org/abs/2404.04125). Since detectors need to know how all LLMs output, we could be doomed. Article suggests there's an upper limit, so maybe not. I'm rooting for you.
@NormalOperator This is one of the reasons I'm looking for third party tools to supplement what we're doing and make it more reliable. There are limits to what we can do alone.
@clarkesworld I am both fascinated to see how you are dealing with this (you seem to be a few steps ahead of a number of scientific journal editors) and sad to see that you are in this position in the first place.
I suspect that to get good detection ability, you likely need to bend a little and tolerate systems that are trained on AI spam, including such spam partly built by cannibalizing real peoples' works.
I wonder if there is some supplemental information you can ask for along with submissions, that functions a bit like a version of the ubiquitous captcha tests, but adapted more specifically for story submissions. I'm not sure what it would be. Perhaps something that requires non-trivial human thought and effort, to be matched up against the submission itself, that could be used for first-round screening.
@aebrockwell At the moment, spam is submitted manually by individuals. CAPTCHA style methods are good for identifying bots. We do use patterns in spammer behavior as part of our current "suspicion" scoring though.
Add comment