ticoombs

ticoombs, 15 days ago (edited 15 days ago)

The downvotes you can see (on this post) are from accounts on your instance then. As this post is semi inflammatory it is highly likely to have garnered some downvotes.

Edit: I guess I was wrong regarding the logic of how downvotes work when we block them. As the http request (used too?) return an error when responding to a downvote. I’ll have to look at it again. As the only way it was/is 15, is if:

• we kept track of downvotes and sent out the activities notification
• your instance got the notifications from other instances about our post, (which is not how Lemmy works unless I’m seriously misunderstanding it.)

ticoombs, 18 days ago

😢 Yep looks like a bug! I’ll investigate today

ticoombs, 18 days ago

Another bug to do with -beta. I’ll submit a bug report later today (I’m meant to be working atm). @dessalines

ticoombs, 22 days ago

That awkward moment when you are the person they are talking about when running beta in production!

ticoombs, 1 month ago

The way we “force federation” is to tell reddthat to “resolve” the comment/post via the API. It’s a roundabout way to force all of the content to still appear.

It’s interesting that it doesn’t trigger notifications but I’m not surprised it doesn’t and when you think about it makes some sort of sense.

The 5-7 day delay is when the actual comment/post is federated from LW as the “time behind” value is 6.3 days currently (and coming down).

I’m reaching out to one of the other admins who apparently wrote their own federation queue. That means we could return a 200 OK response instantly for everything and work on the queue on our own time.
There is still not that much movement on the dev side as it’s a complex problem and the devs don’t seem to want to develop something like a queue currently. Which I understand as it would be a completely new system and would need lots of testing…

On-top of that, every “jump” in backlog results from some abusive kbin bugs. Kbin has some users who send a like and an unlike constantly. In the order of 50k requests in an hour. They like and unlike a post on LW. LW then has to federate ever like and unlike activity. Which ends up being 50k Activites that need to be sent per-server. So it makes federation lag for everyone.

So we’ve managed to catch-up a bit each day and then some kbin bug causes an issue and then we are now +1 extra day behind.

Hearing about the lack of notifications is disheartening. Is it so hard for a system to have some sort of quorum? :(

Hopefully we get something sorted in the next week or two… Otherwise I might block every vote coming in until we catch-up. (As votes account for over 90% of activities).

Tiff

ticoombs, 1 month ago

Oh for threads.net? Once that happens I plan to re-evaluate how everyone feels.
If at the end of the day people feel strongly enough against it then we will keep it but I hope to open it back up again. As (I think) the defed only affects our communities, not content people post to other instances’ communities.

ticoombs, 1 month ago

Thank you for articulating your thoughts. I always enjoy hearing the views of our users! I too will be looking at how this progresses. The more people we can migrate to the fediverse the better!

Though I can’t recommend getting a LW account 😛, purely because I want everyone on Reddthat. I think if you want to investigate Threads/etc, a mastodon type account would be the best course of action. As following single accounts on Lemmy is… non existent (I don’t count RSS feeds).

ticoombs, 1 month ago

Every bit will be used for copious amounts of coffee! 😜. The old server got really cranky, but this new one is raring to go! (Except for this db/pagination issue, but I think I found a good way to investigate).

Thankyou so much!

ticoombs, 1 month ago

Right? Such a good vibe.

ticoombs, 1 month ago

Glad you could find an intermediate home! Your community is always welcome back if needed.

ticoombs, 1 month ago

That’s a big decision I won’t make without community input as it would affect all of us.

If we purely treated it as just another instance with no history then I believe our stance on it would be to allow them, as we are an allow-first type of instance. While there are plenty of people we might not want to interact with, that doesn’t mean we should immediately hit that defederate button.

When taking history into account it becomes a whole different story. One may lean towards just saying no without thought.

All of our content (Lemmy/Fediverse) is public by default (at the present time) searchable by anyone and even if I were to block all of the robots and crawlers it wouldn’t stop anyone from crawling one of the many other sites where all of that content is shared.

A recent feature being worked on is the private/local only communities. If a new Lemmy instance was created and they only used their local only communities, would we enact the same open first policy when their communities are closed for us to use? Or would we still allow them because they can still interact, view comments, vote and generate content for our communities etc?

What if someone created instances purely for profit? They create an instance corner stone piece of the “market” and then run ads? Or made their instance a subscription only instance where you have to pay per month for access?

What if there are instances right now federating with us and will use the comments and posts you make to create a shit-posting-post or to enhance their classification AI? (Obviously I would be personally annoyed, but we can’t stop them)

An analogy of what threads is would be to say threads is a local only fediverse instance like mastodon, with a block on replies. It restricts federation to their users in USA, Canada and Japan and Users cannot see when you comment/reply to their posts and will only see votes. They cannot see your posts either and only allow other fediverse users to follow threads users.

With all of that in mind if we were to continue with our open policy, you would be able to follow threads users and get information from them, but any comments would stay local to the instance that comments on the post (and wouldn’t make it back to threads).

While writing up to this point I was going to stay impartial… But I think the lack of two way communication is what tips the scales towards our next instance block. It might be a worthwhile for keeping up-to-date with people who are on threads who don’t understand what the fediverse is. But still enabled the feature because it gives their content a “wider reach” so to speak. But in the context of Reddthat and people expressing views and opinions, having one sided communication doesn’t match with what we are trying to achieve here.

Tiff

Source(s): help.instagram.com/…/about-threads-and-the-fedive…

PS: As we have started the discussion I’ll leave what I’ve said for the next week to allow everyone to reply and see what the rest of the community thinks before acting/ blocking them.

ticoombs, 1 month ago (edited 1 month ago)

That’s when US timezones wakes up. We physically cannot accept more than 3 requests per second. Physically being the actual network physical limits ( of 3 x 287ms = 861ms, we used to be 930ms+. The server move got us 21ms closer!). LW generates more than 3 activities per second during US “awake” time zones. So we have a period of 8 hours where we need to catch up.

Like I’ve said in our forcing federation post. There isn’t anything to worry about because we are completely up-to-date on posts and comments because of our sync script.

It’s just the sequential nature of Lemmy. I’m going to test a new container in the next 12 hours which removes the blocking metadata generation from the accepting of activities. That way we can guarantee at least 3 activities a second.

Realistically, that is a minor fix but it won’t help with those graphs in the long term. We will need to have parallel sending, for it ever scale.

On a side note while we were on our old server and were using our forcing federation script, we had it set to 10 parallel requests. It didn’t even worry about it. I saw no increase in server load. Which is good news for the lemmyverse in general, as everyone will be able to accept the new parallel sending without needing to increase their hardware.

Tiff

ticoombs, 1 month ago

That’s what I love to hear! 🎉

ticoombs, 1 month ago

/gif legen-dary

ticoombs, 1 month ago (edited 1 month ago)

but upvotes are synchronized later

Correct. All votes are syncronised eventually.

ticoombs, 1 month ago

How are your issues now? 🧐

ticoombs, 1 month ago

🐊 Snap Snap!