Temporarily blocking activities from kbin.social
What’s the Problem?
Over the past few weeks we have been receiving large amounts of activity requests from kbin instances, primarily kbin.social. The issue is intermittent and the root cause is unknown. It appears as though some process occasionally gets stuck in a loop on the kbin side. For example, a legitimate user may upvote a post in one of our communities, causing kbin to send an upvote action, then an undo action, then an upvote action again, an undo action again, and on and on.
This is not straining lemmy.world’s infrastructure at the moment. However, it does create enough network traffic to be problematic for the Lemmy network in general because these activity requests get passed on to other instances. Some instances have already struggled to keep up with normal traffic from lemmy.world due to how Lemmy processes activity requests (see reddthat.com/post/15383278, for more info). When this kbin issue occurs it multiples our outbound traffic and makes the problem worse. The additional activity requests can easily triple the number of outgoing activities from lemmy.world to other instances.
Short-Term Fixes
As a temporary fix we are currently blocking all incoming activity requests from kbin.social.
Users on that site are effectively “read-only” in lemmy.world communities at the moment.
We are hoping to replace the current block with a rate limit on incoming kbin activities. Unfortunately, it may take a few days before the lemmy.world infrastructure team has time to implement that.
We were initially banning the individual users whose actions were being repeated, with the intention of unbanning them when the repeated requests stopped. However, the problem has persisted, and that game of whack-a-mole is manual and too time-consuming. It also gives the impression that those users had done something wrong, which is not the case.
Long-Term Solution
We have reached out to Ernest, who is the kbin dev and kbin.social admin. So far we have not received a response. Ultimately, this appears to be a bug on the kbin side. Hopefully he will be able to dig into this issue soon and resolve it. Until then, we will try to mitigate it on our end as best we can.
We apologize for the inconvenience. Thank you for bearing with us.
For some recent history regarding this issue please see these comments:
lemmy.world/comment/9081380
lemmy.world/comment/8961882
lemmy.world/comment/9330566
Add comment