Lemmy.ml is blocking all requests from /kbin Instances

I discovered yesterday evening that Lemmy.ml is blocking all inbound ActivityPub requests from /kbin instances. Specifically, a 403 'access denied' is returned when the user agent contains "kbinBot" anywhere in the string. This has been causing a cascade of failures with federation for many server owners, flooding the message queue with transport errors.

This doesn't appear to be a mistake; it has been done very deliberately, only on Lemmy.ml. Lemmy.world and other large instances do not exhibit the same behavior. It also isn't a side effect of the bug introduced in Lemmy 0.18. You can observe by sending the following in a terminal

> curl -I --user-agent "kbinBot v0.1" https://lemmy.world/u/test
HTTP/2 200
[...]

> curl -I --user-agent "kbinBot v0.1" https://lemmy.ml/u/test                                
HTTP/2 403
[...]

> curl -I --user-agent "notKbinBot v0.1" https://lemmy.ml/u/test
HTTP/2 403
[...]

> curl -I --user-agent "placeholder-user-agent" https://lemmy.ml/u/test
HTTP/2 200
[...]

Additional evidence of this not being a Lemmy 0.18 bug:

  • This occurs when making web requests to any location on the Lemmy.ml webserver, not just ActivityPub endpoints.

  • Go to https://fedidb.org/software/lemmy and pick an instance running 0.18.0. Perform the above commands, replacing the URL for Lemmy.ml with that particular instance's address.

If this continues, my instance may need to defederate from Lemmy.ml. This is especially problematic because Lemmy.ml continues to federate information outbound to other kbin instances while refusing to allow inbound communication from them.

Spoofing the user agent is less than ideal, and doesn't respect Lemmy.ml's potential wish to not be contacted by /kbin instances. I don't post this to create division between communities, but I do hope that I can draw awareness to what's going on here. Defederating /kbin instances entirely would even be better than arbitrarily denying access one-way. This said, we should all attempt to maintain a good-faith interpretation until otherwise indicated by the Lemmy developers. It's possibel that this is a firewall misconfiguration or some other webserver-related bug.

Relevant comment from me (#354 - [BUG] Critical errors/failed messages during messenger:consume)

Edits:

  • Yes, people have already tried reaching out to the Lemmy instance admins in their Matrix room with no answer.

  • Someone has posed a question on Lemmy.ml about the block here: https://lemmy.ml/post/1563840

ernest,
ernest avatar

It's possible that this is a consequence of the latest Lemmy update, in which a lot has changed. I have noted that kbin has some issues with request signature in communication with certain instances. I will try to check it tomorrow first thing in the morning.

Kaldo,
Kaldo avatar

I've noticed we are also not federating with lemmy.wtf for some reason, maybe it's something related. They are connected well with other lemmy instances but kbin can't get a hold of them despite being on their list of linked instances

melroy,
@melroy@kbin.melroy.org avatar

Sorry to tell you this, but Lemmy.ml seems to actively blocking 'kbinbot' user-agent requests...

svovl,

Sorry to go off topic here, but it seems that no messages are coming in and out from kbin.social.

See, e.g., https://kbin.dk/m/kbinMeta@kbin.social/t/3054/Lemmy-ml-is-blocking-all-requests-from-kbin-Instances and https://kbin.melroy.org/m/kbinMeta@kbin.social/t/2560/Lemmy-ml-is-blocking-all-requests-from-kbin-Instances

LollerCorleone,
LollerCorleone avatar

Those links are working fine. Kbin is federating well with those instances. A bit of latency is normal.

svovl,

No upvotes, comments or boosts go through

r00ty,

It takes time. I just setup my own instance and I sent a comment from there, it took 40 minutes to arrive on kbin.social, upvotes and replies have not made it back to my instance yet some 45-60mins after they happened.

!deleted168346,

deleted_by_author

    •
    r00ty,

    Because, when you post here from kbin.social any other instance with kbinMeta@kbin.social will get a copy of that and vice-versa. But each side is exchanging posts from multiple magazines to multiple other instances. It's also balancing resource usage for people visiting the site too.

    Also new instances are gradually fetching the back-catalog of posts for various magazines (and communities on lemmy). So all of this leads to a delay.

    Anecdotally the delay is quite short this morning. Yesterday it reached up to 2 hours from my view at least.

    r00ty,

    Case in point, it took less than a minute for this to reach my instance (this is me, posting from my instance... Maybe I should have used another username... This one has a picture, is the instance me).

    !deleted168346,

    deleted_by_author

    •
    r00ty,

    Well, there's a bit to unpack here. When you download something in seconds that's your whole connection to a server that is on a super fast connection.

    Most people running instances are on a much more modest combination of hardware and connection and even the bigger ones (kbin.social etc) are not going to have a connection as fast as dedicated download CDNs can offer. I would expect they probably have a gigabit, and at most 10gbit. That's shared between everyone on the site downloading cat pictures, posting, refreshing AND the federation of all the new content to and from other instances.

    But that's really not the problem here at all. Far more of a load is the processing of the incoming and outgoing messages to and from many instances. This takes CPU load (and to a lesser extent memory), and this is shared between the message queue for these inter-instance messages, the web server and database.

    When you look at how the fediverse of kbin and lemmy is laid out. You will see there's a handful of larger instances with most of the popular magazines/communities. This means that they're doing the lion's share of this processing. Couple the fact that the population has exploded over the last month or so, even these larger instances might be struggling with hardware and/or infrastructure layout (maybe running all on one box, and needing to split the load for example). That's speculation though.

    At any rate, for whatever reason (maybe the lemmy problems backing up message queues with errors) things were MUCH slower last night.

    melroy,
    @melroy@kbin.melroy.org avatar

    Still after 5 hours.. the upvotes are behind.. I hope AP will scale well in the future.. This since looks not great.

    YMS,
    YMS avatar

    But the oldest comments both there and on kbin.social are more than 55 minutes old without appearing in the respective other place.

    LollerCorleone,
    LollerCorleone avatar

    Until kbin.social servers are fully upgraded, such delays will occur. Even afterwards, it might happen if the other instance is running on slow servers. We are still in the early stages of this platform, and there is no huge corporate throwing money at us (it is so early that there hasn't even been an actual 'Release' yet). These quirks are to be expected as kbin develops.

    Upvotes_Kills_Birds,
    Upvotes_Kills_Birds avatar

    Wow, thank you for the quick communication. Amazing aptitude. Almost nowhere else do you find a lead dev in the comment trenches letting us know what's happening, I'm kinda baffled.

    riktor,
    riktor avatar

    we love ernest!

    phi1997,

    If they don't want us to communicate with the users of their server, they should be defederated. Sucks for the users, but they can hop to another server whose admins don't break communication inelegantly and ghost admins who want to know what's going on.

    If it's just a bug, then obviously we should wait for it to be fixed, but if they're not even saying they'll look into it, I don't have much hope.

    ripcord,
    ripcord avatar

    Before assuming, has anyone...asked them?

    barista,

    Relevant question on Lemmy.ml, in addition to prompting on Matrix is here: https://lemmy.ml/post/1563840

    IntlLawGnome,
    IntlLawGnome avatar

    Yeah, I posted that to !fediverse from my lemmy.ca account. I figured that if the admins aren't answering questions via Matrix or whatever, maybe someone there would have an answer. (So far, nothing more than the same kind of speculation that's happening here.)

    phi1997,

    According to the edit in the main post, yes, they have been asked.

    Deceptichum,
    Deceptichum avatar

    Before assuming no one has asked them, let’s read the post

    “Yes, people have already tried reaching out to the Lemmy instance admins in their Matrix room with no answer.”

    ripcord,
    ripcord avatar

    Ok, but, as the thread over there says, it's not like they're singling out kbin as far as the code shows. Just because no one there has answered yet doesn't mean they've refused to answer. Give it some time or some actual replies before assuming the worst.

    Also that edit was added after my question.

    It's been a few hours, this is way way too fast for the pitchforks to come out.

    Deceptichum,
    Deceptichum avatar

    Code and config is different, it’s irrelevant if it’s not in the git.

    frex, (edited )

    If I use a firewall to block kbin in user agent then it doesn't matter what code is running, it will throw 403.

    Edit: Just checked out, the first request with kbin user agent, it doesn't have x-powered-by: Express, suggesting that it is blocked in nginx level, it doesn't even reach the backend Express. This will not be reflected in the code whatsoever.

     ~  curl -I --user-agent "notKbinBot v0.1" https://lemmy.ml/u/test
HTTP/2 403
server: nginx
date: Wed, 28 Jun 2023 01:50:53 GMT
content-type: text/html
content-length: 146
vary: Accept-Encoding

 ~  curl -I --user-agent "placeholder-user-agent" https://lemmy.ml/u/test
HTTP/2 200
server: nginx
...
x-powered-by: Express
...
    melroy,
    @melroy@kbin.melroy.org avatar

    Yea this is bad. Lemmy.ml is actively blocking on 'kbinbot' (case insensitive) string in the user agent request.

    $ curl -I --user-agent "this is KBINBOT" https://lemmy.ml/u/test
HTTP/2 403
....

$ curl -I --user-agent "this is KBINBO" https://lemmy.ml/u/test
HTTP/2 200
....

$ curl -I --user-agent "this is BINBOT" https://lemmy.ml/u/test
HTTP/2 200
....
    svovl,

    In the spirit of a thriving fediverse, I suggest not to assume the worst of other parts of it before we know a bit more of what's going on.

    Zzinck,

    Lemmy is no different than Reddit. They silence and ban anyone they don't agree with and do not care about the users.

    ripcord,
    ripcord avatar

    You're making one hell of an assumption here.

    Most likely it is a misconfiguration or bug. Or misunderstanding.

    Narrrz,
    • assert "no different"
    • list one single, disputable similarity
    r00ty,

    I don't think it's assuming anything to say that they deliberately blocked user agents containing the text "kbinbot" (case insensitive). I think it's fairly clear that's exactly what has happened here. Instructions to test as many variations as you want yourself are right in the main post in the thread.

    Now, what we cannot say is whether it was done with malicious intent. There's some plausible reasons they might do it that are not malicious.

    ripcord,
    ripcord avatar

    I think it's assuming a lot to say the things I replied to.

    r00ty,

    I was more replying to the second paragraph. It's a pretty deliberate configuration choice on the web server.

    mnejing,
    mnejing avatar

    Ideally it's a config error at the firewall. I saw an interesting idea posed on the lemmy post suggesting that it may have been targeted by a DDoS that used kbinbot in the user-agent string.

    Ultimately, it's not happening at a code level, it's absolutely happening at a firewall level (nginx, which, for those who don't understand, is kind of acting like the door lock on your apartment building, where you need to go through the main security before you can get to your own place. Sort of the same idea here). I just spent a bunch of time testing a bunch of various user-agent strings, and it very specifically is matching "kbinbot". No wildcarding within the word, but it a string like "blahbinbotblah" will 403, whereas "blahkbbinbotblah" won't (and various other forms, like k.bin.bot, or kasdfbinasdfbot.)

    It's pretty specific. Anyone who deals with firewalls in any capacity understands how nginx works, and specifically why they and others are raising an alarm.

    So yeah, ideally it's a misconfiguration, otherwise it's a fairly clear message.

    assbutt,
    assbutt avatar

    deleted_by_author

    •
    melroy,
    @melroy@kbin.melroy.org avatar

    People are indeed getting ghosted from Lemmy's Matrix channel, when they report this issue.

    Tyrannosauralisk,

    There is a question about this up on lemmy.ml here:

    https://lemmy.ml/post/1563840

    btaf45,

    Hey Yote.zip is saying this. What's up with that?

    " I’ll add that I have recently noticed that practically nothing that I post has been getting through to kbin."

    AnonymousLlama,
    AnonymousLlama avatar

    At least they're equally as confused, hopefully this is just an issue in communication and the federation of content will be back in full swing soon

    Eisenhowever,

    Doesnt surprise me, the developers of lemmy (which are owners of .ml) have an agenda. They are into censoring on ukraine news, and other stuff. I dont get why people are choosing lemmy over kbin when they are equally bad looking

    TheRoarer,

    Wait why is kbin bad looking?

    TwilightVulpine,

    It looks better than reddit.

    slicedcheesegremlin,
    slicedcheesegremlin avatar

    yeah, kbin is probably the nicest looking reddit alternative i've seen. Really sleek design without leaning into the overly overly mobile-focused watered down New Reddit bs, while also not nearly as hideous as old reddit.

    mochi,
    mochi avatar

    I do wish kbin looked a bit better on mobile, though. I have larger font on my iPhone and instead of wrapping, the text just goes off the screen and can't be viewed.

    artisanrox,
    artisanrox avatar

    IMHO Kbin actually looks better on my mobile than on desktop.

    I have both set to the aqua/greenish theme and on mobile my upvotes/interactions show up bright yellow...but on desktop there's no difference. Even if I take off "follow desktop personalization theme". (firefox for both.)

    mochi,
    mochi avatar

    I'm glad you think so. Please see the attached screenshot for an example of what I'm talking about. I'd like to have the text one size bigger so I can read it more easily, but then whole words are cut off or missing.

    Eisenhowever,

    Its a huge downgrade from reddit in terms of visual quality. But im soeaking on mobile safari so i guess that will explain alot but anyways heres what i see.

    The posts that have pictures dont really show that well until you click on it. You need to scroll down a whole thread to comment on a post. Creating a post/thread is confusing, its hard to navigate “magazines” (idk why theyd call it that), theres a lot of problems with it.

    Lemmy looked pretty bad on mobile safari, far worse than kbin

    UrbenLegend, (edited )

    Eh? What are you talking about? There's literally a bunch of Russia and Ukraine news at https://lemmy.ml/c/worldnews. This rumor mill needs to stop. People started some bad rumors about Lemmy devs just because of their political association and now its blown up into full lies.

    I use both Lemmy and Kbin, but I prefer Lemmy for a bunch of different reasons:

    1. Interface is cleaner and more performant.
    2. Getting to your list of subscribed magazines in kbin is a pain in the ass. Why is it buried in Settings?
    3. Kbin votes are public. This is just bad for privacy. It's one thing to say that votes are public due to federation, and whole other thing to blatantly show it in the UI. There's a reason why voting at a government level is kept secret, just sayin'.
    4. Also why the hell are they called magazines? None of the names in kbin map to existing conventions. "Threads" are posts. "Magazines" are communities or groups. I never get used to it because it isn't natural to call these things the way kbin calls them.
    Bodleum,

    I think liking an internet post and voting for the leadership of the government are not suitable for comparison here.

    Eisenhowever,

    Sounds like youre spewing a lot of bullshit or intentionally misguiding. Read the thread below you and youll see.

    assbutt,
    assbutt avatar

    deleted_by_author

    •
    Onii-Chan, (edited )
    Onii-Chan avatar

    This guy acting like those in charge at Lemmy being pro-CCP and censoring dissenting opinions isn't a valid reason for not wanting to be involved with them. Fuck Lemmy. Fuck authoritarians.

    EnglishMobster, (edited )
    EnglishMobster avatar

    I wouldn't really call it "rumors" when you can go directly to the horse's mouth:

    https://www.reddit.com/r/communism/comments/cqgztr/fuck_the_white_supremacist_reddit_admins_want_me/

    https://web.archive.org/web/20230626055233/https://old.reddit.com/r/communism/comments/cqgztr/fuck_the_white_supremacist_reddit_admins_want_me/

    Hey all, longtime Marxist-leninist, recorder of left audiobooks, and megathread shitposter here.

    Posting this in light of a recent one week Reddit ban I earned for shitting on US police, as I'm sure many of us have gotten in recent weeks.

    So I've spent the past few months working on a self hostable, federated, Reddit alternative called Lemmy, and it's pretty much ready to go. Unlike here we'd have ultimate control over all content, and would never have to self censor.

    Obviously as communists, we agitate where the people are, so we should never abandon Reddit entirely, but it's been clear to all of us from day one, that communities like this stand on unsteady ground, and could be banned or quarantined at any moment by the white supremacist Reddit admins. This would be both a backup and a potentially better alternative. Moderation abilities are there, as well as a slur filter.

    Raddle isn't an option obviously since it's run by this arch anti tankie scum, ziq.

    I wanted to ask ppl here if they'd like me to host an instance, and mod all the current mods here.

    My concern is with this line: Obviously as communists, we agitate where the people are. I'm pretty left-leaning myself (I draw the line at authoritarianism though), but they're very open about using their platform to push an agenda.

    That post was made by the founder of Lemmy and the instance they mention at the end became Lemmygrad (because lemmy.ml and Lemmygrad are the same people - the ".ml" in "lemmy.ml" even stands for "Marxist-Leninist").

    With Lemmy.ml as the "main" instance, you're exposing all these posts to an admin team who has openly said "we are agitating for our views wherever the people go, and we have ultimate control over all content." They have already removed posts critical of China as being "orientalist" (which shows a severe lack of understanding of what "orientalism" even is - it's not "anything that criticizes our Dear Leader"). I can't trust that any community hosted on lemmy.ml is free of bias.

    It isn't "rumors" when you can clearly go and back it up with a source.

    (FWIW - I don't mind Lemmy as a platform. My Lemmy account is actually from 2020, before I realized what they truly were. The maintainers have done a good job of keeping politics out of the software, and the concept of only using "ethical" things is a myth when your computer has rare earth elements in it. As far as instances go, Lemmy.world seems fantastic and has grown to be far larger than the tankie instances. But Kbin is more fully-featured and has a lot more long-term potential, even though it's lacking in the short-term.)

    static,
    static avatar

    Any news on this? It's till broken.

    crowsby,
    crowsby avatar

    They're moving to a paid API as a prank.

    camelbeard,

    At first I thought it was maybe a config error to block bots. But after some testing I see that the 403 is only given when the word "kbinbot" is in the user agent. String that just return the normal response I tested are "testbot", "kbinbo", "binbot".

    Jajcus,

    Maybe it is some kind of automated rule 'block bots that cause lots of traffic'. kbin is becoming more and more significant, so it probably causes some non-insignificant load on Lemmy servers. Legitimate load, but the IDS rules would not know that.

    AnonymousLlama,
    AnonymousLlama avatar

    I think the approach of changing the user agent in the interim is a good solution, if there's a technical reason they don't want kbin traffic they can be adults and reply in the chat, not just sneakily blocking incoming traffic

    Cloudless,
    Cloudless avatar

    This will save me time. I am trying to block all communities hosted by the tankies instance.

    curiosityLynx,

    You can block the instance itself, you know. No need to hunt for the communities.

    assbutt, (edited )
    assbutt avatar

    deleted_by_author

    •
    tal,
    tal avatar

    I don't believe that there is currently a link to the /d/DOMAIN page -- that it needs to be manually-entered. That makes the functionality hard to find, and probably should have a PR to add a link somewhere in the UI.

    curiosityLynx,

    I'm sorry I wasn't all that helpful. IIRC you need to find a post by someone on that community (or in that community?). Anyway, the name of the instance will be displayed on the card of that post and is clickable, which will bring you to the kbin page about the instance, where you can block it.

    luna,

    Happy to see this. I've been avoiding .ml anyway, this makes my job easier. Fuck the Lemmy devs.

    btaf45,

    It looks like this is all intentional. It is a good thing that this all came out now and not later. lemmy.ml has admins that are worse than beehive.

    https://lemmy.ml/post/1167199

    !deleted168346,

    deleted_by_author

    •
    Gull,

    Outraged feelings aren't the problem. Nobody is complaining about defederating Nazis. Defending and lying about Stalinist atrocities is morally no different from defending and lying about Nazi atrocities. If you let Nazis and tankies overrun your communities they won't be a "real alternative." Just like with Nazis, tolerating intolerance is intolerant.

    Pamasich,
    Pamasich avatar

    The issue is, it's only "kbinBot" and variations of that that don't work. If you just use "bot" itself, everything works fine. So it's not just an overreaction to the bot problem, unless all bot instances impersonated kbin.

    The more likely answer is that lemmy.ml wanted to exclude kbin users from participating, but without actually limiting the freedom of their own users (which defederation would do).

    And on a relatively unrelated note, splitting the already minuscule fediverse, will kill any chance for it to become a real alternative.
    Even if Lemmy's dev political views can be outrageous to some people, defederating them would be the beginning of the end imo.

    The current situation is misleadingly confusing to the average user and kbin moderators can't do their job properly.
    I don't see how defederation, especially when temporary until the issue is fixed, harms anyone when we already can't interact properly anyway. I think the current situation is more harmful. Imo this sounds like the kind of situation defederation exists for.

    heyfluxay,

    This explains why I could subscribe to their Firefox sub/bin/whatever

    Rnway,

    Well, checking this post from my Lemmy instance (lemmy.sdf.org), it appears to be in sync with what I see from my Kbin instance (fedia.io).

    Does that imply that whatever issue was in play here has been fixed?

    Cloudless,
    Cloudless avatar

    This issue is specific to the lemmy.ml instance. So lemmy.sdf.org should not be affected.

    If the lemmy devs start blocking kbin by adding code, it would be time to fork the code.

    sharkfeek,
    sharkfeek avatar

    I've noticed this too. While I can post on a lemmy.ml community from kbin, I can't see it show up on my lemmy account.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • kbinMeta
  • DreamBathrooms
  • thenastyranch
  • mdbf
  • Durango
  • Youngstown
  • slotface
  • hgfsjryuu7
  • vwfavf
  • rosin
  • kavyap
  • osvaldo12
  • PowerRangers
  • InstantRegret
  • magazineikmin
  • normalnudes
  • khanakhh
  • GTA5RPClips
  • ethstaker
  • cubers
  • ngwrru68w68
  • tacticalgear
  • everett
  • tester
  • Leos
  • cisconetworking
  • modclub
  • anitta
  • provamag3
  • All magazines
    •