Google indexes a discussion page using content from other pages

[handymenny]

Bug Report

Current Behavior

Google's crawler indexes page X of a discussion (discussion/?page=X) using also the text/images from other pages. This means that what Google's crawler sees doesn't always match what users see once they open that page.

Example

Google "ok, ma io ho fatto questi test a scopo informativo/illustrativo site:forum.fibra.click"

Second result is page 12, but that post belongs to page 10

Expected Behavior
Google's crawler should only look at posts on a specific page, i.e. in the above example Google should link to page 10

Environment

• Flarum version: 1.2.0
• Website URL: https://forum.fibra.click/

Possible Solutions

1. (Hacky) Disable javascript for all crawlers (limited to discussions)
2. Disable posts auto-loading for crawlers, adding a button "load more" that points to the next page
3. Disable posts auto-loading for all users, adding a button "load more" that enables posts auto-loading

[matteocontrini]

I also notice that in the second result in the screenshot the number of posts shown is wrong. It says 20 but that's not the total number of posts in the discussion.

[askvortsov1]

(2) seems like a reasonable solution. The button already exists, so we'd just need to add a userAgent check as to whether a given user is a bot or not before triggering autoload.

Another solution could be updating the canonical url as the page scrolls, but:

• I'm not confident that the indexing engine would take that into account.
• We'd need to duplicate page title and canonical URL generation logic across the frontend and backend.

I'll also note that there's a bit more duplication than I'd like, as it seems like search engines have both the old post number-based links and the new page-based links stored. But that should get resolved naturally over time.

I also notice that in the second result in the screenshot the number of posts shown is wrong. It says 20 but that's not the total number of posts in the discussion.

This one has me a bit stumped. I'm not sure where exactly Google is getting this information; it doesn't seem to show up on search results for NodeBB or Discourse communities, and from a few quick Google searches I'm not seeing whether there's a meta tag we could use to provide accurate information on this. @Hari-Bonda @jaspervriends or anyone else knowledgable in SEO, any ideas as to where this is coming from?

[davwheat]

Not sure if the SEO extension embeds JSON-LD, but that might be it?

[matteocontrini]

Not sure if the SEO extension embeds JSON-LD, but that might be it?

The extension has an option for listing the posts in JSON-LD but I haven't enabled it for performance reasons (I'm the owner of the forum of the example above). Also, I'm pretty sure that the posts count and the other metadata have been there even before the extension existed, so I guess it's just Google magic.

[Hari-Bonda]

i am aware of this issue since Aug 2021, we had to disable indexing subpages (posts) until this issue gets resolved so we went with Discussion Canonical URL extension as a temporary solution https://github.com/SychO9/flarum-discussion-canonical-url

Coming to the solution.

When flarum is breaking discussions into pages for example

page=?near1
page=?near2
page=?near3

you should change the page title too (test discussion page 1 of 2) & (test discussion page 2 of 2)

this title was implemented in flarum v1.2 and using linguist I have made a few tweaks to get the page title output what I need

the second most important thing is when the sum of posts are considered as page page=?near2 , for that particular page flarum should maintain the first post (in case of second page 21st post) content as page description.

let us say if the second page is displaying the 20th post as the first post for the second page you should display the 20th post content as a page meta description for that page if you fail to do this you will end up seeing duplicate titles and description for all pages which is a huge SEO mistake google will consider as you are trying to fool the spider with duplicate content and gets confused to rank the main discussion page.

i don't know how JSON or JS is working but maintain a different DB or something to get the data like I have mentioned.

i am not an expert but if you observe WordPress pages you can easily notice this.

[karl-bullock]

I ran into this from a different direction (a user reported duplicated posts when fetching discussion pages with curl) and found two server-side mechanisms that produce this symptom in the rendered HTML, independent of the frontend auto-loading discussed above. Both are in the no-JS render, which is the initial HTML a crawler receives.

1. A numbered URL declares one page as canonical and renders a different one.

Forum\Content\Discussion computes $page as 1 + intdiv($near, 20) and uses it for the previous/next links and for the canonical URL, but the window it renders is centred on the linked post. Measured on a 1.8 forum with a 30 post discussion:

URL	canonical	renders	page 2 actually is
/d/1/25	?page=2	posts 15-30	posts 21-30

So six of page 1's posts are served under a URL declaring itself page 2. That is the same direction as the report in this issue, earlier content appearing on a later page.

2. flarum/mentions leaks quoted posts onto the page.

The renderer picks what to print by scanning $apiDocument->included for anything with type === 'posts', a discussion relationship and contentHtml. flarum/mentions adds posts.mentionedBy together with posts.mentionedBy.discussion, so every post that quoted a post on the page satisfies all three conditions and gets printed as if it belonged there. On a busy public 1.8 forum, the first eight pages of one discussion rendered 29, 20, 23, 24, 21, 23, 20 and 24 posts instead of 20, with 21 posts appearing on two different pages.

Worth noting for triage: this second one leaks in the opposite direction. Every one of those 21 duplicates put a later post onto an earlier page, because mentionedBy means "posts that quoted this one". So mechanism 2 does not explain the page-10-under-page-12 example in the original report, while mechanism 1 does.

PR: #4852 fixes both on 1.x by asking the posts endpoint for the page and rendering its primary data, so extensions cannot contribute posts to it and the rendered window always matches the canonical URL. It has integration tests for both, and 2.x already works this way.

I would not call that a fix for this issue, since the auto-loading path discussed above is a separate mechanism and may well still contribute. But it removes two concrete ways the served HTML ends up holding another page's posts, and both are measurable without involving a crawler.