Bug: Sitemap generates duplicate language paths (/en/stable/en/) regardless of html_baseurl configuration

[BillSousa]

Details

• Read the Docs project URL: https://readthedocs.org/projects/pybear/
• Read the Docs username (if applicable): https://readthedocs.org/profiles/BillSousa/

Project: https://pybear.readthedocs.io/

Environment:
Sphinx Version: 9.1.0
Sphinx Theme: pydata_sphinx_theme
Python Version: 3.12

Problem:
All sitemap URLs contain a doubled /en/ path, making them invalid 404s:
https://pybear.readthedocs.io/en/stable/en/index.html ← broken
https://pybear.readthedocs.io/en/stable/index.html ← correct
Root cause identified:
RTD injects -D language=en into the Sphinx build command:
python -m sphinx -T -b html -d _build/doctrees -D language=en . $READTHEDOCS_OUTPUT/html
This causes Sphinx 9.x to output files into an en/ subdirectory, making page links resolve to en/somepage.html. Combined with html_baseurl, this produces doubled paths in the sitemap.
Confirmed:

RTD's native sitemap has the same bug regardless of sphinx-sitemap being present or absent
Setting html_baseurl to either https://pybear.readthedocs.io/ or https://pybear.readthedocs.io/en/stable/ does not fix it
Actual pages load correctly at the right URLs
Canonical tags on pages are correct

Impact:
Google Search Console flags these pages as "Duplicate, Google chose different canonical than user" and drops them from the index because the sitemap directs crawlers to broken paths while canonical tags point to the correct URLs. This causes steady decline in indexed pages over time.
Requested Fix:
Update the sitemap generation to normalize URLs so that platform-level language/version strings are not duplicated regardless of html_baseurl or Sphinx environment variables.

[humitos]

All sitemap URLs contain a doubled /en/ path, making them invalid 404s:

Taking a look at https://pybear.readthedocs.io/sitemap.xml, I don't see the doubled /en/ path you are mentioning. Can you expand on your report and how we can produce it?

This causes Sphinx 9.x to output files into an en/ subdirectory, making page links resolve to en/somepage.html. Combined with html_baseurl, this produces doubled paths in the sitemap.

Does this only happens on Sphinx 9.x? Does it work on previous versions?

[BillSousa]

Taking a look at https://pybear.readthedocs.io/sitemap.xml, I don't see the doubled /en/ path you are mentioning.Can you expand on your report?
The reason you don't see the doubled /en/ path in pybear is because on 7/12/26 I created a post-build hook that runs after readthedocs finishes building the docs and the sitemap. The hook is in https://github.com/PylarBear/pybear/blob/main/docs/generate_sitemap.py. The hook looks for paths containing the double /en in $READTHEDOCS_OUTPUT then if it finds one, it overwrites that path with the correct path in sitemap.xml. If the builds were run without the hook you would see the double /en paths in sitemap.xml.

How we can produce it?

1. Create a Sphinx project on RTD with default settings.
2. Trigger a build on RTD.
3. Check sitemap.xml — URLs will contain /en/stable/en/pagename.html
4. In the build log, I observed that RTD injects -D language=en into the Sphinx build command, which I believe causes Sphinx to output files into an en/ subdirectory, producing the doubled path in the sitemap.

This causes Sphinx 9.x to output files into an en/ subdirectory, making page links resolve to en/somepage.html. Combined with html_baseurl, this produces doubled paths in the sitemap.
Does this only happens on Sphinx 9.x? Does it work on previous versions?
I ran the builds with (sphinx==9.1.0 + sphinx-sitemap==2.9.0) and (sphinx==8.1.3 + sphinx-sitemap==2.6.0); both of them produce the double /en/ paths. I also confirmed the bug exists without sphinx-sitemap entirely — RTD's native sitemap produces the same doubled /en/ paths.