Move crosspost detection to backend

[Nutomic]

There have been some complaints that crosspost detection is not working well:

• https://lemmy.ml/post/44346952
• https://lemmy.ml/post/44397015

The problem is that it is handled in the frontend, and only works if both posts are on the same page. By moving it to the backend and handling it in SQL, we can merge crossposts even if they are on different pages.

Related: #6262

[dessalines]

The GetPost endpoint does return cross-posts, but not GetPosts. This is because it'd be way too costly to search the post table for every result.

The problem is that ppl are pointing out issues for other apps like piefed voyager, that don't do any front-end cross-post simplification. Its already working in lemmy-ui.

I think this should be a no-fix just for performance reasons, and also because front ends would want to handle feed duplication in their own ways (or not at all).

[Nutomic]

The issue in the first link is on lemmy.ca, problem is that crossposts arent deduplicated across different pages.

Its not something we need to actively work on, but worth keeping an issue open. Maybe after some years, someone will have an idea how it could be fixed while avoiding performance problems.

[dessalines]

The issue in the first link is on lemmy.ca, problem is that crossposts arent deduplicated across different pages.

That's true, front end de-duplication is only going to work for the first page. Its also probably not that useful for infinite scrolling clients either.

The only somewhat decent perfomance way to do it off the top of my head, is to

• Call GetPosts
• Collect up the post urls of the results
• Search for dupes of those post urls using post::url.eq_any
• Create a map from post url to PostView, removing all but the first result and putting them in a post_view.cross_posts dummy array.

That at least only requires 2 queries, but still could be costly depending on the performance of that eq_any. It also could potentially mess up paging because we could be removing same-page results.