Feature suggestion: duplicate posts handling

Hello LJDawson, let me start with saying how happy I am to have Sync back :)

The one issue I have with Lemmy is duplicated posts between different communities (see screenshot). Posts usually have same title, author and link, yet differ in community/server they are posted to. It would be great to be able to make the appear as one item on the feed with some note below that links to both instances’ posts, or something like that basically to make them not appear twice or thrice one under another.

mojo,

Yess, I hide read posts, but tons of duplicates appear in my feed anyways. This is probably a bigger problem and will be even bigger with scaling on Lemmy. So it’ll be interesting how we solve it. Maybe group together federated posts that post the same link in a 24 hour period? You could open a post and it’d say something like “also posted in X” and you can see all the similar posts. This would be a lot of work on Lemmy’s end too though. Just food for thought.

marsara9,
@marsara9@lemmy.world avatar

Playing devil’s advocate for a bit… So these are just cross-posts. Which existed even on Reddit. …I assume they weren’t handled in any way in Sync or Reddit?

But let’s say this is fixed… What to do about the multiple comments threads? How would you reconcile them with each other? Especially since the user can choose different ways to sort the comments as well. Would all of this logic normally handled by the Lemmy back-end now need to run on your phone? Also how do you choose which post / instance to actually display and which ones to hide?

Btw, I’m not trying to dismiss the idea. Just want to call out some of the technical problems that might come up trying to implement such a feature. As well as ask questions to try and determine exactly how such a feature is expected to work.

fiestorra,

Yeah, I also think this is more of a Lemmy problem, rather than sync one.

Even if they are called the same at the end of the day they are different communities hosted on separate instances.

burtek,

Just because it’s a Lemmy problem, doesn’t mean Sync can’t do anything about it, for it’s user’s benefit.

drzoidberg,
@drzoidberg@lemmy.world avatar

It’s fine to dismiss the idea, because on reddit you could see the same user post the exact same article on a dozen relevant subs. If you try to hide them, which one do you hide?

It’s a stupid idea, akin to wanting to ban all reposts.

zalack,
zalack avatar

Have the various comment threads on a carousel once you click in. Because of the fractured nature of Lemmy servers I feel like I see way more reposts than on Reddit. It would be nice for them to get merged in some way.

murtaza64,

Since this issue is most prevalent while browsing new, you could temporarily store (a hash of) the titles and contents of each post during a given browsing session, and if a post matches another post on the same page (e.g. one set of posts from Everything/new), hide it and allow the user to see comments from each instance of the post when they view the canonical (earliest, or most active, or returned first) post.

If another duplicate post is later loaded (through scrolling) the post could be hidden and replaced with a smaller indicator telling the user a duplicate post has been detected, and they can click it to view the comments.

burtek,

I would imagine one post preview with a very compact list of instances on which the posts were posted so that you can enter either of those posts’ view.

Or merge the posts into one view, show somehow it’s a merged post, and merge comments as well (and let you choose the instance on which you want to comment if you create a new top-level comment).

Reddit had similar issue but it wasn’t that bad since you hardly ever had 2-4 similar communities - something that appears to be very common with decentralised lemmy.

eek2121,

There are a few ways to handle it, but yes it would either need to be done client side or via a server proxy.

I am actually toying around with something (that I may or may not release publicly, depends on how far I get before I lose interest) to help app developers do stuff like this without heavy processing on the client.

stu,

I feel like this should only be a problem when users don’t make use of the crosspost functionality built into Lemmy (which ultimately should only be a technical problem that Lemmy clients need to address at the time of post creation). Aside from appropriate options in a client that should allow you to crosspost to multiple communities at once, I feel like this is a server side issue more than a client side one because the server shouldn’t be returning duplicates like this if posts are appropriately crossposted IMHO.

burtek,

Well, the comments are separate for each crosspost so I’d expect the server to return all crossposts equally…

stu,

I don’t really know the best way to deal with that on the server API side, but I don’t think it’s to return them all as separate posts in the main feed. This is a problem that could clearly get much worse before it gets better because it could make the All/New feed really painful to read.

coconutxyz,

I’d imagine there will be rules to match the user, post title etc. Not sure if that will be an easy implementation

burtek,

I’m not saying it’s easy, just that it’d be nice to have.

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • syncforlemmy@lemmy.world
  • DreamBathrooms
  • magazineikmin
  • everett
  • InstantRegret
  • rosin
  • Youngstown
  • slotface
  • love
  • khanakhh
  • kavyap
  • tacticalgear
  • GTA5RPClips
  • thenastyranch
  • modclub
  • megavids
  • mdbf
  • normalnudes
  • Durango
  • ethstaker
  • osvaldo12
  • cubers
  • ngwrru68w68
  • tester
  • anitta
  • cisconetworking
  • Leos
  • provamag3
  • JUstTest
  • All magazines
    •