I have a question for #SFBA members and the broader #Fedierse & #MastoAdmin community. This isn’t as an SFBA admin, just a curious fedi user. We arn’t crafting policy from this.
How would you feel about an algorithm/AI/system scanning an instances feed and filing reports based off the code of conduct, with context and links for moderators to review? With proper disclosures on the about page and a good effort to make it known.
Note, this would only be reporting. The computer would have no mod powers
@cd24 I guess that's a pretty broad category of things. I think so long as there is a human making decisions and it is otherwise privacy preserving it could be helpful. You would want to make sure that whatever is doing the scanning does not store any posts longer than needed for reporting or use them for other purposes without the user's consent, and is transparent about that. Some things would be pretty safe to look for, like flagging an account that is new and sending a ton of messages
@cd24 one of the most refreshing things about Mastadon is seeing people expressing real thoughts and sharing knowledge, not ads or promotions. Lately, I have noticed more and more bots, influencers and trolls - communication gets less clear as we grow.
Ultimately, I trust your (plural w/mods) judgement, especially if it’s a quality of life tool that gives you more time to make the site enjoyable for you. It seems odd to be okay with AI presence to preserve human voices but why not.
@cd24 My first inclination was to say "no" because I have a knee-jerk negative reaction to anything "AI."
But on reflection I think this would probably be fine, at least on the local timeline, and maybe beyond. I want moderators to have the tools they need; it's a hard job!
@cd24 I don't think I have a philosophical objection (although I might change my mind because the question a little bit broke my brain) but I do have questions about whether (a) there would be too many false positives for this to be useful, and (b) whether moderation decisions which are repetitive enough to be easily machine classified are a symptom of something else, like lack of community norms or rapid growth. Probably other questions which are even less clearly formulated in my head.
@soaproot the principle use case in my mind is spam. Especially as LLM systems grow, there’s a huge opportunity for automated spam (like the .social attack earlier), and I’m wondering what we could do to harden our spaces here!
Efficacy is always a critical metric, and I’m sure if something were to be made it would take several iterations to tune.
@cd24 Definitely would support this. I do a version of this sometimes where I'll go through and just find objectionable accounts to block. It would be great to have an algorithm tag crypto accounts and spam.
@cd24 how is this different from actual people in the instance reporting the bad accounts? Admins will have to check anyways. Since this system doesn't have any automation does it really make a difference? How about creating a system where users that have been reporting bad accounts get a score to make their reports higher in the list? I think that might work better in the long term.
@cd24 admins can add confidence points to accounts that have accurately reported bad accounts to make their work easier. If it's not the case the account loses points to make them less truthful.
@klizana this isn’t really the problem. Our report queue is often empty, or nearly so. It’s more that I’d be interested in understanding how we could use a machine system to help digest the massive federated timeline streaming by at 1000000 miles a minute (or the local timeline which is also really speedy!)
But point well taken - I agree the user base by and large does a great job of this! And I’m quite happy with how wonderful SFBAs members have been at highlighting issues
@klizana there’s also the case where users encounter highly offensive material in comments. If at all possible, I would want to sequester extreme, hateful, or graphic replies before a user encounters them. Not after they file a report.
@cd24 make sure it's not filing reports against marginalized people using sluts used against them. this is always the first bad take that comes out of these systems.
@tyr this is one of the many reasons I would never wire something like this up directly to the mod tools. I doubt a computer could know this 100%, and I’d want a person to make the call
To be clear, there’s nothing in the works, or even planned. Just musing.
There wasn't an option for "maybe" so I chose the closest thing. Automod could be useful in some circumstances, for example, a sudden influx of spambots; or a time when no one on the mod team is available. I think it needs three features:
• Could be turned on or off at the discretion of the mod team
• Instead of just flagging problematic posts, it would hide them pending review
• Parameters it uses to determine which posts are problematic can be modified by humans
@cd24 Nextdoor has a Leadbot that does something like this. It's terrible. While it does bring posts to our attention, it almost always misses the mark -- it flags a good many acceptable posts by one neighbor, but what she's replying to is the problem. It's easy to "keep" hers but we end up having to read the whole thread and report the real violations.
Granted, we'd otherwise miss the problematic posts if Leadbot didn't do this, but it just isn't that smart and it feels like more work for us because of that.
Done properly, it might be a good thing. But if not, I'd almost prefer it weren't there at all.
@cd24 such a tool implies that there's recurring, obvious bad behavior that's not being flagged by users. in which case, sure. there just needs to be a clear benefit to timely discovery that outweighs maintenance cost and the extra work from the poor signal/noise ratio of such bots. an occasional reminder to the user base of flagging tools might be as effective and cheaper
@pootriarch absolutely! I think one of the super powers of SFBA is that people here file good reports when they see stuff that’s up (and my co-mods do excellent work in responding to them!)
There’s not really a specific tool or use case - but it would absolutely need to be evaluated for efficacy.
@cd24 I'll warn that I've gotten waaaayyyy to thinky in an academic and existential sense about the whole concept of 'AI' lately, and I'd like to see us stop using that damn term entirely.
If someone were to create a moderation tool, optional but available, that very, very clearly a) documented its algorithmic basis and b) absolutely could not act or be corrupted to act on its own, I'd consider reviewing and trying out such a thing ...
@cd24 ... were I to get to a point where my content management and user report handling were overwhelming the team.
I attended a talk today that had a speaker repeatedly returning to some core ideas, though, one of which was "slow down" and another of which was "what for?" or "why?" when it comes to such tech. And I don't mean "why make that tool" being answered with "to deal with more <something> faster", it was "why do you need that accelleration?
@cd24 Why do you need to LOOK for that trouble? Is it actually serving the meat serfs of the community to do so? Is PRE-EMPTIVELY identifying problematic content really needed, when what we could be doing instead is cultivating a local but direct democracy which makes these identifications collectively by their reporting and their dialogue?
A huge downfall of the twits and faceborks is they have decided FOR ME what I may encounter, before I even know some things are out there. That costs me.
@Johannab from a philosophical perspective I totally agree! I think mods should have a measured and proportional response - and banning is an extreme measure.
That said, I am most interested in the practical. A spam attack creating thousands of accounts would put an unreasonable burden on the servers and could create service problems - I figure a system like this would be more useful for that case.
@cd24 We've had such things happen at least 3 times in recent weeks that I personally encountered. Different sort of specifics each time. Each was handled by the human collective, the worst was disruptive for a couple of hours.
Could it happen again and escalate? Yes. Could we build better frameworks (code and policy) and community plans to PROactively reduce likelihood and damage, rather than decide we need enhanced, automated, post-hoc surveillance? I think also yes. Admittedly, not easy.
Add comment