crashdoom

crashdoom, 1 month ago

Case 1: Non-Interactive Maintenance Poll

crashdoom, 1 month ago

Case 1 - Overnight (10 PM to 8 AM)

crashdoom, 1 month ago

Case 1 - Evening (5 PM to 10 PM)

crashdoom, 1 month ago

Case 1 - Daytime (8 AM to 5 PM)

crashdoom, 1 month ago

Case 2: Interactive Maintenance Poll

crashdoom, 1 month ago

Case 2 - Daytime (8 AM to 5 PM)

crashdoom, 1 month ago

Case 2 - Evening (5 PM to 10 PM)

crashdoom, 1 month ago

Case 2 - Overnight (10 PM to 8 AM)

crashdoom, 1 month ago

US Mountain Time! (I put that on the other posts, but forgot here, my bad!)

Maintenance will typically be as needed, but I think I’ll need to make a new poll for capturing weekend / weekday preferences, I hadn’t thought about differentiating that.

crashdoom, 3 months ago

#2: I’ve had some light experience before specifically with TensorFlow Lite models during my degree program. For the Coral Edge TPU, we wanted to off-load the processing to try to get the speed as near to zero latency as possible, though admittedly, it would potentially be superfluous. I’m also looking into some existing models I could potentially use but hadn’t found any that particularly stood out, but if anyone has any recommendations I’d love to check them out!

#3: Good question; If the system flags a post automatically as potentially spam, and the team determine it’s not spam, I would probably like to be able to train on that message as “ham” / not-spam to avoid future false positives. But, that would be an extension of the scope of what we’d train on, so I’d very much like feedback on that too.

#4: Yes, when a user is limited the profile will show a content warning before the contents of the profile. I believe the prompt is something like “This user has been hidden by the moderators of [instance name]”. For repeated mis-identifications, yes, edge-cases like this we could approve the user and exempt them from future automated reports.

crashdoom, 3 months ago

Appreciate the feedback so far, let me try to see if I can answer most / many of the questions:

What are the risks of #4?

Many users are worried about the risk of automated actions going wrong and not knowing what we mean with “pattern of spammy behavior.”

For how we would identify the pattern of behavior that would allow for automated actions, we would review any major spam wave, such as the one we’ve been experiencing over the past few days:

https://pawb.social/pictrs/image/c5f00a6a-8868-4d93-a675-7ff194f1652f.pnghttps://pawb.social/pictrs/image/2aa3aa48-c8c2-42fc-8e7d-f26ffd21ec19.png

We would then identify any indicators we could use that are indicative of the known spam, and create a heuristic ruleset that would limit or suspend those accounts while targeting only those accounts actively engaging in the spam, not just referring to it. There are additional safeguards we can add, such as preventing rules being applied to users where the user is followed by someone on our instances.

For the risk of automated actions going wrong, if we were using a limit (not a suspend) then the account would be hidden from public view but could still be viewed if specifically searched by name, it would also suppress all notifications from that user unless they are followed by you. (e.g. if they messaged you out of the blue, you wouldn’t see it if you weren’t following them.)

If a suspend was used, the account would be marked for deletion from our instances but all follower relationships would immediately break (e.g. if you were following them, the system would automatically unfollow when they are suspended). Typically, we can restore data within 30 days, but follower relationships are typically unrecoverable. So long as rules are appropriately limited in scope to only target those with a lot of spam indicators, no false-positives should occur.

What about appeals?

For local users (anyone registered on furry.engineer and pawb.fun), all actions against your account (except reports) can be appealed. If you have a post removed or are suspended, all actions can be appealed directly to the admin team.

For remote users, we can remove restrictions on remote accounts if we receive an appeal from any of our users, or by the affected account directly. This can be done via email, or just through a DM to one of the admins who can pass it to the team.

Would the AI model have oversight?

Yes. Where the team believe the filter has flagged sufficient content appropriately and maintains no false-positives, we may promote a model or ruleset to allowing automated actions (limit / suspend).

We’ll keep an eye on the actions of each ruleset by reviewing the daily / weekly actions taken to ensure they meet the criteria and have not misidentified any users or content, and we’ll also start publicly tracking the statistics of the models / rulesets we create and use, including a count of false-positives or reversed decisions.

Will you notify users?

Due to limitations in Mastodon, we can only notify local users (users on furry.engineer or pawb.fun) when actions are taken against their account; This process happens automatically when your post is removed, or your account is warned, limited, or suspended.

There’s no easy way to notify remote users other than sending them a DM, but doing so could be seen as spammy or lead to inciting further abusive behavior by informing them of our activity. While we can have transparency with our users due to having an invite-only platform, other instances are frequently open-registration which can allow the abusive user to re-create an account to continue to harass our users. BUT, I’m open to suggestions on this.

crashdoom, 3 months ago

So, the issue lays in that there’s no technical way to notify the remote user (someone not on furry.engineer or pawb.fun) that they’ve been suspended on our end, without sending a message to them directly. If we suspend them on our end, that doesn’t per se suspend them on their end and they wouldn’t know that their messages were no longer reaching our users; They would still be able to message other users on their instance, and users on other instances, but not to our users.

We’re apprehensive about notifying remote accounts specifically because we don’t often know the moderation practices of the remote instance (to know if they’ll deal with it, or if they have open-registration allowing anyone to join without approval) and it may encourage further abusive behavior through ban evasions (creating new accounts on that instance or elsewhere to continue messaging) from the user being made aware that we’re no longer receiving their messages.

crashdoom, 3 months ago (edited 3 months ago)

Is this working again? We were experiencing some networking issues, but they appear to be resolved (phiresky.github.io/lemmy-federation-state/site?do…).

Edit: Does appear that we’re lagging a little on out-going federation activities due to the volume of incoming requests. I’ll look into setting up horizontal scaling for Lemmy in the coming days once we’ve finished setting up the final node in the Kubernetes cluster! We’re currently hamstrung for resources on the two servers we have.

crashdoom, 3 months ago

@Kovukono @name_NULL111653 @liquidparasyte Does this still appear to be occurring for you? We were having technical issues over the past day but they appear to be resolved now.

crashdoom, 3 months ago

When it next occurs, could you email network@pawb.social and attach a screenshot of the error page? If it happens in an app, could you let me know what action you’re trying to do and I’ll see if I can pin it down in the logs! :3

crashdoom, 3 months ago

We’re aware of this issue and are working to resolve it. Yesterday, the last of the files from Cloudflare R2 transferred and we’re working on getting the files backed up before removing unnecessary files, and restoring the missing files to the local storage. We’re needing to do the removal of unnecessary files as there are ~6TB of stored media, but only 1TB of used media between furry.engineer and pawb.fun.

crashdoom, 2 months ago

Apologies for the delay. We’re working on this over the course of the weekend trying to restore the emojis.

crashdoom, 3 months ago

Looks really, really well done! :D

crashdoom, 3 months ago

Didn’t even think to do that! Added to the sidebar :3

crashdoom, 3 months ago

Yes, we’ll be maintaining:

• Multiple replicas across different disks (local)
• Hourly and daily snapshots (local)
• Regular off-site backups for disaster recovery

crashdoom, 3 months ago

Status update

Both instances are back online again! We’re currently transferring cached media from remote instances to the local storage, so avatars, emojis, and older attachments may currently appear as broken images.

As of Feb 7th at 10:45 PM Mountain Time, pawb.fun has re-generated all feeds, while furry.engineer is continuing with an estimated 25 minutes to go. We’re also re-generating the ElasticSearch indicies which power the full-text search system and expect that to continue through the night.

crashdoom, 3 months ago

Aware and investigating!