Disallow package docs except for latest

[oraNod]

The purpose of this change is to disallow crawlers from all Ansible package docs versions except for the latest version, currently 13. We want to do this so that search results return for latest versions.

[samccann]

lgtm but we could go all the way up to Ansible 10 (and soon Ansible 12).

[samccann]

lgtm

[oraNod]

@samccann I've been rethinking disallowing the supported package versions.

Consider the RTD docs about version states: https://docs.readthedocs.com/platform/latest/versions.html#version-states

The default behaviour is to disallow hidden versions in robots.txt. Only active versions are indexed.

We need to explicitly declare whatever rules are needed because I don't think subprojects get their own robots.txt and we're using a custom one. But I think we should consider allowing any of the versions that are in the flyout widget on RTD, which follows the same pattern as the supported branches in the source repo.

[oraNod]

Here are the versions on the flyout widget:

This corresponds to the branches in https://github.com/ansible/ansible-documentation/blob/2aada29638a9ad769e3a8c7d66868a4408e0d9b7/.github/workflows/pip-compile-dev.yml

@samccann What do you think about hiding the devel build on RTD? That build will still be available but do we want devel in the flyout if we're disallowing it in robots.txt?

[oraNod]

Maybe if we hide devel in the flyout menu we can also remove this from the banner:

You are reading the devel version of the Ansible documentation - this version is not guaranteed stable. Use the version selection to the left if you want the latest (stable) released version.

[gundalow]

Not to make this more complex, though I agree we should think about /devel/ vs /latest/.
I could extend my RTD Bulk Fix script to point to /latest/ on all links that exist. Once case I can think we might want /devel/ is links to roadmaps.

I've marked this PR as draft till Don had decided what he wants to do here.

[oraNod]

Not to make this more complex, though I agree we should think about /devel/ vs /latest/. I could extend my RTD Bulk Fix script to point to /latest/ on all links that exist. Once case I can think we might want /devel/ is links to roadmaps.

I've marked this PR as draft till Don had decided what he wants to do here.

I'm happy with these changes now. We can disallow crawlers/indexing for everything in the package docs except for latest. The versions will still be available in the RTD flyout menu where humans will look for them.

[oraNod]

On a side note I'm also thinking about adding a disallow rule for projects/ansible-core/* but we can do that in a follow on PR because it is a separate project.

[samccann]

For devel (whether to have it in the flyout) - this imo should be a community decision. I Think either way the banner needs to stay. Even with the website saying devel, people would confuse it with /latest/ and thus we added the banner.

[oraNod]

For devel (whether to have it in the flyout) - this imo should be a community decision. I Think either way the banner needs to stay. Even with the website saying devel, people would confuse it with /latest/ and thus we added the banner.

Yeah, that's fair enough. I can start a thread in the forum. What about disallowing devel in robots.txt though? Do you think we should hold off?

[gundalow]

Is there anything left, or can we merge this as-is?

[oraNod]

Let's just merge this. I think we should also just keep devel in the version switcher. Probably the only people who will look at that will be folks in the SC etc. And we will still have the banner.

Thanks for the reviews @gundalow and @samccann