What's up with "The magazine from the federated server may be incomplete. Browse more on the original instance."? Is this working as intended or is it a bug?

dannekrose, 10 months ago

@Treedrake

Most replies here are correct. To clarify and summarize:

1. The source of truth for a magazine/community is the server name that appears after the magazine name.

e.g. kbinMeta@kbin.social <--- the source of truth is kbin.social.
2. ActivityPub and the Fediverse is a "Push" model. What does this mean?

Imagine subscribing to a real-world newspaper or magazine with home delivery(few these days will actually remember this, but try to imagine at least). You will get all new issues delivered to you from the moment you became a subscriber, but you don't get copies of all the newspapers or magazines they have ever printed delivered to you. You only get things moving forward. That's the same with the Fediverse. After you subscribe or follow something, you will get all the new content moving forward, but not what has been created so far.

3. To extend #2, it's a "push once" model. What does that mean? It means that if I create content from my instance (which is not kbin.social) to the magazine kbinMeta@kbin.social, my content will get pushed to kbin.social. Kbin.social, however, will not "re-push" that content to everyone that kbin.social knows is subscribed to the magazine.

So how does my new content that I created in kbinMeta@kbin.social show up on other instances that are not kbin.social? I thought you said your content only gets pushed once?

Correct. However, it's not quite as simple as my instance pushing just to kbin.social. Strictly speaking, (and this is based on experience with other platforms, not specifically how kbin works since I haven't verified this for kbin 100%) when I create the content, my instance will push to kbin.social and all other instances (not users) that my instance knows are also subscribed to specifically kbinMeta@kbin.social. So my instance actually knows a subset of the instances that are subscribed to kbinMeta@kbin.social and will push the new content to each of those other instances. My instance, however, won't necessarily know all the other instances that are subscribed to kbinMeta@kbin.social. As a result, some instances won't see my new content because it wasn't pushed to them.

3. As a result, to let users know about this potential gap, not only does it mean that older content doesn't automatically show up, it also means that not necessarily all new content will show up either.

Note on #3: I haven't fully verified this. This statement is based on how other, non-kbin instances handle federation. This is how "likes" work across platforms like Mastodon, Calckey, etc. I see no evidence (yet) that this is any different for kbin.

Treedrake, 10 months ago

Thanks for the clarifications. Regarding 3, why does some new content from e.g. kbinMeta@kbin.social gets pushed to an instance like lemmy.world, but not all new content? I mean, if some content gets pushed it seems to be aware of that instance. Or is it random which content gets pushed? Does this mean that the other instance will never get all new content? This seems like quite the issue?

Edit: ok, I think I get it. So if someone from a random instance posts to e.g., technology@beehaw.org, and that instance doesn't know about kbin.social, it won't get pushed to this instances representation of the technology@beehaw.org magazine?

dannekrose, 10 months ago

@Treedrake

Edit: ok, I think I get it. So if someone from a random instance posts to e.g., technology@beehaw.org, and that instance doesn't know about kbin.social, it won't get pushed to this instances representation of the technology@beehaw.org magazine?

Correct. It's only "pushed once" by the instance that the creator's account is on. Of course it will push to the source of truth, but it will only push to other instances it knows are also following that magazine since it doesn't necessarily know all of the instances that follow that magazine. In your example, yes, if the creator of the content (which in this example is an account not on beehaw.org) posts to technology@beehaw.org, their instance will push to beehaw.org and others, but if it doesn't know that kbin.social has any followers of the magazine/community, it won't push to kbin.social.

One other possibility is that the 3rd-party instance does know about kbin.social (for example), but has blocked (defederated with) kbin.social OR kbins.social has blocked that 3rd-party instance, my expectation would be that such content won't show up on kbin.social's "copy" of the community.

Treedrake, 10 months ago

So it's basically working as intended? It does feel like an annoying "issue" though. Thanks for explaining anyhow!

dannekrose, 10 months ago

@Treedrake

You're welcome!

Yes, it is working as intended. The idea is that each instance is responsible for pushing content once, then it's the responsibility of the receiving instance to process and display the content to the relevant users/accounts.

As a side note, if everything was "re-pushed" out, the load becomes even more on the "source of truth" for larger communities with wide federation and a lot of new content generated locally and remotely. I could see this being leveraged to take down servers by simply spamming really large communities (with large federation) with small content forcing the "source of truth" to now "re-push" the content to every server that is knows about for every single new comment, or reply, or post.

Treedrake, 10 months ago

Fair enough! I guess it's a downside that comes with the federation.

Kichae, 10 months ago

Eh, it's not much of a downside in an active community. The model is push once at time of publication, there are other systems in place that republish content and help it propagate through the network.

On kbin, there's a "boost"... well, it's not really stylized as a button, but there's a "boost" feature, which acts to republish posts or comments and allows them to reach newer subscribers. I'm not sure how it works on Lemmy, as there's no explicit boost, but there are mechanisms there to renew active back content.

It all just means we, as users and members of communities, need to take on a little bit of responsibility for ensuring that interesting or valuable content propagates.

Prootje, 10 months ago

If i understand it correctly, the system is built to spread info and decrease load but at the same time to get the best and most up to date info everyone needs to be on one instance?

Weird, must be a mistake in my understanding of the fediverse

melroy, 10 months ago

This is a "feature", I think backfilling would be still a nice to have with kbin.

ZILtoid1991, 10 months ago

This kind of an issue with Mastodon-Misskey interaction too.

Shogun, 10 months ago

This was something I noticed as well. All of my subscriptions across other instances are showing this message. At first they were pretty bare showing a handful of posts. After a day or two they seem to be picking up most new content.

Its confusing because I still don't know if kbin is displaying all new content? What about old content does that eventually sync up as well?

Xepher, 10 months ago

I'd like to know exactly how this works as well. I've seen people say there's no real downside to joining a smaller kbin instance as you can still get to other magazines in different instances, but this seems like it would be a drawback if you can't browse old threads in a remote magazine. Or would it eventually pull all the threads in? Would be nice to know what the expected behavior is.

Chozo, 10 months ago

I've noticed that sometimes new threads from federated magazines sometimes don't show up for me in Kbin, as well as older ones. I think there may be some sort of syncing issue that causes that, perhaps Kbin refreshes data from other instances at set intervals or something.

Treedrake, 10 months ago

Yes, this is the issue I'm referring to!

dannekrose, 10 months ago

@Treedrake

One thing that also plays a role is the size of the queues on the different instances. Kbin.social may get things pushed to it, but with queue sizes reaching 500 thousand or more, it takes time to process them.

@Xepher @Chozo

Kaldo, 10 months ago

I think it's actual data loss, not just lengthy queue times. I see new threads and comments appear but for some of the older threads, even days after they were active, I see missing comments if I compare it between kbin.social and the original instance truth.

Silfeed, 10 months ago

From what I’ve come to understand, which I am no expert, I think it means that the posts from those magazines start when someone on kbin subbed to that particular magazine. I had found a pretty small mag on Lemmy that only had maybe 5 posts when going to the original instance but after I subbed to it I would see nothing on kbin. They made 1 post after I subbed to it and that is the only post to show up on kbin, even to today even though it’s been a week or so since I subbed to it.

I had read somewhere on here that when it comes to the individual communities, kbin will only federate with that magazine after one person on here subs to it, after that anyone in kbin will see posts from that day forward.

Again, I could be very wrong, I’m just guessing based on my experience.

TLDR: the magazine only shows posts from when someone subbed forward so it will always be incomplete, I think.

Treedrake, 10 months ago

I've heard this as well, but this is regarding new posts and comments not showing up. Not the old ones. And when they do show up after some unknown time, it might still be incomplete.

Teppic, 10 months ago

No, once somebody (anybody) from your instance has subscribed, all posts from the community from that point on 'should' mirror. The warning is in relation to older posts.
That said several of the Lemmy servers have been struggling with load over the last week or two (and kbin.social has occasional downtime too). Occasionally some posts might not push out to all other instances properly.

8BitFriendly, 10 months ago

Good question. Magazines that mention this often display a different number of posts compared to the original instance, just as the message says.
I also wonder whether this will be fixed or is there to stay because 100% synchronization is not achievable.

LexaPrime, 10 months ago

I see it as a just-in-case message, to let you know that you're browsing a non-kbin community and that some federation features may break on either side, even if they are not broken at the moment. There may also be some differences in how posts and comments are sorted between kbin and the original instance, and - I'm not sure if that's exactly how it works - potentially you may see different comments depending on who the two instances federate with.

Treedrake, 10 months ago

I think the main issue is that I'm not getting all comments or posts from the other magazine, even though all the posts and comments are from that same magazine (but on the other instance). There might be some delay involved and that it sooner or later will retrieve them as older posts seems to have more comments on them.

Pamasich, 10 months ago

Keep in mind that some instances might just not be federating with kbin. If they did in the past, then I imagine their magazine remains available on kbin but no new posts or comments will appear from the defederated instance. Not sure if that's how it works though, just my assumption.

In particular, I don't know if lemmy.ml is still blocking kbin, but don't be surprised if their posts won't appear.

Treedrake, 10 months ago

Yes, but this is regarding magazines from e.g. lemmy.world, which to my knowledge is not defederating from kbin.