Add support for big sitemaps (> 50K urls)

[incognitozen]

What version of Hugo are you using (hugo version)?

$ hugo version
hugo v0.97.3+extended linux/amd64 BuildDate=unknown

Does this issue reproduce with the latest release?

Yes,

If you create a sitemap on a site with over 50K urls , Google complains that the file is too big.

Your Sitemap contains too many URLs. Please create multiple Sitemaps with up to 50000 URLs each and submit all Sitemaps.

I looked at the docs and noticed that there is no way to override this. Technically not a bug, but this makes it difficult to submit sitemap to Google.

[sifigi4335]

See https://discourse.gohugo.io/t/feature-request-sitemapindex-for-sitemaps-with-50k-links/33214

[incognitozen]

HI @carerragt

Thanks for pointing in the right direction.

Please note davidsneighbour response on that thread.

Often requested, but technically not possible.

This means that there is a reasonable demand and need for this feature. I have a site with a single 'type'. I don't have categories,tags or other taxonomies. There are 79K url's in my site all belonging to the same type.

Hence, Ju52 proposal may not work with me as I don't have different sections on the site. I understand that this may not be the top priority but it is a problem worth fixing. I run hundred's of sites all in wordpress. Atleast 50%+ sites would have more than 50K url's.

[sifigi4335]

Perhaps what you should be asking is how to split the sitemap list to multiple files. The 50k is a Google limitation, not Hugo's per se.

[incognitozen]

hi @carerragt

Sure, I understand.

It does beg the discussion that the very notion of having a sitemap is to submit to search engines. Without this need, there is no requirement for a sitemap. Both Google and Bing that provide consoles for managing the sitemap submission specifically request a sitemap that is chunked over 50K.

I would open a forum thread but if you solicit community feedback from those that have larger site, they will tell you that this might be a very important feature for them.

[midzer]

I also have a site with 50k+ pages in a single sitemap.

Adopting some kind of automatically splitting due an external limit in Hugo might break things for others. We should complain about the limit at the external search engine at first. Maybe those provider can up the limit to let's say 100k?

[incognitozen]

@midzer

You can certainly try but there is a rational for them to limit the file to 50K url being the size of the file.
Try downloading a file that has 50K url and the size will be approx 4MB.

Furthermore, Google and Bing certainly don't need to change their processing pipelines because a static site gen decided that sitemap.xml shouldn't be split. If Hugo wants to be adopted, then the onus of adding features or making changes inline with industry expectation lies with Hugo and not other providers.