Improve SEO of website

[larshinueber]

The findability of the website on search engines is quite bad. In my case, the (seemingly) only term that returns the website as first result is "tudat space", whereas any other term links to either an unofficial docs page ("tudat installation" gives https://aleixpinardell.github.io/tudat/installation/index.html as first result) or Github pages ("tudat tle propagation" gives Tudat/tudat#103).

It also seems that the only subpage showing in the results is the landing page (for some reason linked as https://tudat.tudelft.nl/ instead of https://docs.tudat.space/en/latest/), but none of the user guide pages or examples can be found.

This has lead to misleading references of users in the past and should be improved (how?).

Some resources that might be useful:

• https://docs.readthedocs.com/platform/stable/guides/technical-docs-seo-guide.html
• https://docs.readthedocs.com/platform/stable/reference/robots.html
• https://developers.google.com/search/docs/crawling-indexing/url-structure (in particular dashes vs underscores). For redirects, see https://github.com/nextstrain/readthedocs-cli

[jo11he]

I agree we have an issue with visibility of our key resources.
I propose to define three key tudat resources:
a) tudat landing page (https://docs.tudat.space/en/latest/)
b) tudat API reference (https://py.api.tudat.space/en/latest/)
c) source code (https://github.com/tudat-team/tudat-bundle)

I believe at the very least we should get all three key resources on the first page (ideally with landing page being top result) of any conventional search engine when using generic prompts such as:

• "tudat documentation", "tudatpy documentation", "tudat docs", "tudatpy docs"
• "tudat astrodnamics toolbox", "tudatpy astrodnamics toolbox"
• "tudatpy api", "tudat api"

My experience with the searchability of these key resources is the following:
Our most important resource (a) is basically invisible. The api reference (b) is more visible, but still not as searchable as we should like. The source code (c) is typically also not a top result, instead the tudat-team home page on GitHub (not bad...), the tudat-team/tudatpy source code repo or developer docs show up.

I think there are two measures to be taken to improve this:

1. Weed out legacy / irrelevant websites that pop up during searches. That could already significantly improve things. That could include asking users/developers to make their forks of tudat repos private, as well as logs from readthedocs and old/legacy tudat homepages and developer guidelines. I think we may also have some redundancy between the tudat-team landing page and our tudat landing page (a) and we should consider resolving that in order to streamline the engagement with our resources. Below, I will provide a list of urls that frequently pop up when using the generic prompts specified above, which could serve as a starting point for the process of weeding out.

2. Optimize our key resources. As Lars suggests above, there is some work to be done in optimizing our resources search engine performance. That may be more work intensive and it may be worth defining a "desired" behavior in response to certain prompts to set up the SEO properly. I also think the role of having two names for the software could be an issue in this context... using tudat in search engines tends to give me the better results (even with regards to tudatpy resources) and I think it has to do with the fact that in text, headers, repos etc we use the term tudat more often that tudatpy.

List of urls as starting point for weeding out (may have some redundancy):

https://tudat-developer.readthedocs.io/en/stable/primer/docs/multidoc.html
https://tudat-team.github.io/tudatpy/rotation_model.html
https://github.com/tudat-team/tudat-multidoc
https://readthedocs.org/projects/tudatpy/
https://www.tudelft.nl/library/support/library-voor-onderzoekers/onderzoek-starten/dcc/showcase-projects
https://aleixpinardell.github.io/tudat/
https://tudat-developer.readthedocs.io/en/latest/index.html
https://tudat-developer.readthedocs.io/en/v0.1.1/
https://github.com/tudat-team/tudat-space
https://readthedocs.org/projects/tudat-space/
https://aleixpinardell.github.io/tudat/installation/index.html
https://indico.esa.int/event/224/contributions/3891/attachments/3170/3912/ICATT_2018_-_TUDAT_overview_20181107_vPrint.pdf
https://app.readthedocs.org/projects/tudatpy/

[larshinueber]

Fully agree with your comments :)

Regarding the pages you listed:

https://tudat-developer.readthedocs.io/en/stable/primer/docs/multidoc.html: Should be deprecated and removed once all contents are moved to the wiki pages
https://tudat-team.github.io/tudatpy/rotation_model.html: This was a deployment in the tudatpy repo, I believe this is removed now.
https://github.com/tudat-team/tudat-multidoc: Should be archived at some point, hopefully this will lower it in the search ranking
https://readthedocs.org/projects/tudatpy/: I dont think we can do much here, other than improving our own SEO :)
https://www.tudelft.nl/library/support/library-voor-onderzoekers/onderzoek-starten/dcc/showcase-projects: Same here, not much we can do other than improving our own SEO
https://aleixpinardell.github.io/tudat/: As per Dominic's request, this was taken down
https://tudat-developer.readthedocs.io/en/latest/index.html: See above, should be deprecated
https://tudat-developer.readthedocs.io/en/v0.1.1/: See above, should be deprecated
https://github.com/tudat-team/tudat-space: Again not much we can do, at least this links to the correct website in the sidebar
https://readthedocs.org/projects/tudat-space/: RTD page, not much we can do other than improving our own SEO
https://aleixpinardell.github.io/tudat/installation/index.html: See above, was taken down
https://indico.esa.int/event/224/contributions/3891/attachments/3170/3912/ICATT_2018_-_TUDAT_overview_20181107_vPrint.pdf: Ideally this would be superseded by a proper journal publication, e.g. in the Journal of Open Source Software
https://app.readthedocs.org/projects/tudatpy/: RTD page, not much we can do other than improving our own SEO

In summary, I think the problem is mostly on our side. The fact that literally no page shows up in the search results other than the main landing page (not that other pages are just ranked bad, they dont show up at all) makes me think there is a wrong configuration or systematic mistake we have on our side.

My suspicion is something related to the robots.txt file that prevents search engines from indexing the website properly or the unfavorable link structure using underscores instead of dashes. But I'm not an expert on SEO by any means, so any input is welcome :)

[DominicDirkx]

I've archived the developer docs and multidoc page.

I can't find a way to take down the developer docs page temporarily, but I have been able to download the current state of the latest build (attached as website). With this, I think we can delete the developer docs once we have everything relevant migrated (?)

tudat-developer-readthedocs-io-en-latest.zip

[larshinueber]

Great! Since all the source code can still be accessed in the archived repos, I think we can delete the RTD pages.
In the developer docs RTD page, you should be able to go to Configure version:

and then switch the Active flip, which should remove the build:

[DominicDirkx]

Yep, I tried that, but I can't make the default version 'Inactive' and there always needs to be a default version :)

[DominicDirkx]

But, since we have the source, and a built version of the page here if we need it, I'll go ahead and delete the RTD page

https://tudat-developer.readthedocs.io/en/latest/index.html

Done!

[larshinueber]

@DominicDirkx @jo11he Would you mind trying to search for tudat/tudatpy using any of the terms mentioned in the comments above?
I tried various search engines and it seems that all the deprecated websites are removed from my search results.

The fact that literally no page shows up in the search results other than the main landing page (not that other pages are just ranked bad, they dont show up at all) makes me think there is a wrong configuration or systematic mistake we have on our side.

Unfortunately this is still the case, but at least all the search results point to the current versions of our docs :)

[DominicDirkx]

When I search "tudat documentation" or "tudatpy documentation" I get:

https://docs.tudat.space/en/latest/
https://py.api.tudat.space/en/latest/
https://github.com/tudat-team/tudat-space

as first, second and third hit :) When searching "tudat docs" the first two hits are the same, the third one is the (archived)

https://github.com/tudat-team/tudat-multidoc

Searching "tudat astrodnamics toolbox", "tudatpy astrodnamics toolbox" gets me to

https://docs.tudat.space/en/latest/
https://www.tudelft.nl/digital-competence-centre/projects/tudat
https://github.com/tudat-team/tudatpy
https://research-software-directory.org/projects/tudat

Searching "tudat api" and "tudatpy api" gives me the API as first hit

All searches on Google in Incognito mode. Overall looks pretty good!

[DominicDirkx]

And, when searching just 'tudat', the video from @luigigisolfi (https://www.youtube.com/watch?v=hGlaJi66vzU) is hit number 8 :)

[larshinueber]

Awesome! That's pretty much the same results I get as well 👍

One thing that I would still like to improve is searching for more specific terms. E.g. I would expect that "tudat acceleration models" should return https://docs.tudat.space/en/latest/user-guide/state-propagation/propagation-setup/translational/available-acceleration-models.html or https://py.api.tudat.space/en/latest/acceleration.html but at the moment it only returns an issue from the old Tudat organization, the 2018 ICATT paper, the linear sensitivity analysis example etc.
Similarly, "tudat coupled translational rotational dynamics" being very specific should return https://docs.tudat.space/en/latest/examples/tudatpy-examples/propagation/coupled_translational_rotational_dynamics.html but at the moment returns the 2018 EPSC paper (good, no complaints here :) ), an old issue form Tudat again, physics stackexchange etc.

[luigigisolfi]

I noticed that, when googling "tudatpy examples", I am never able to find a link to the corresponding page , and I always need to pass by the main tudat landing page

[larshinueber]

I noticed that, when googling "tudatpy examples", I am never able to find a link to the corresponding page , and I always need to pass by the main tudat landing page

Exactly, I think this is the same underlying issue as with the more specific search terms. It seems that only the top-level docs.tudat.space website is indexed, any of the subpages you cannot find in search engines.