Migration of QLever-backed Scholia to Toolforge

[Daniel-Mietchen]

The current Scholia setup is

• https://scholia.toolforge.org/ with a full-graph real-time Blazegraph backend currently scheduled to be switched off on January 20
• https://qlever.scholia.wiki/ with a full-graph real-time QLever backend, both hosted on servers of the University of Freiburg

We want to migrate this such that the Toolforge-hosted version is using a full-graph real-time QLever backend hosted on Wikimedia servers, for which we have a test system (wikiqlever) running on Wikimedia Cloud Services.

In order to get there, I suggest the following steps:

• Set up a copy of the current https://scholia.toolforge.org/ at https://scholia-blazegraph.toolforge.org/ #2767
• Set up a Toolforge copy of the QLever-based Scholia at https://scholia-qlever.toolforge.org/ #2768
• Set up a Toolforge copy of the QLever-adjusted Wikidata Query Service UI #2774
• Switch https://scholia-qlever.toolforge.org/ to using wikiqlever as the QLever backend #2769
• Test https://scholia-qlever.toolforge.org/ for traffic-related issues #2770
• Set up copies of wikiqlever as needed, along with some load balancer #2771
• OPTIONAL:
  • Set up copies of wikiqlever as needed, along with some load balancer #2771
• If https://scholia-qlever.toolforge.org/ is stable, move its code to https://scholia.toolforge.org/ #2772
• keep https://scholia-qlever.toolforge.org/ for testing purposes and perhaps as a staging server
• keep https://scholia-blazegraph.toolforge.org/ for documentation purposes as long as Wikidata is still maintaining its Blazegraph endpoints
• keep https://qlever.scholia.wiki/ for testing purposes and as a backup in case of Toolforge issues
• Set up some mechanism that can redirect traffic from https://scholia.toolforge.org/ to https://qlever.scholia.wiki/ if needed #2773
• closely monitor the setup for some weeks and tweak as needed.

This is part of

• [epic] Scholia transition to a QLever backend #2756

[Daniel-Mietchen]

I have started the scholia-blazegraph and scholia-qlever projects on Toolforge:

[physikerwelt]

OPTIONAL: set up copies of wikiqlever as needed, along with some load balancer

I think this will require some time, so we should try to run a load test on wikiqlever early:

1. For one, we need to request additional resources, and
2. I have never set up LB before.
3. Also, Wikiqlever is currently missing streaming updates, which I would like to integrate before scaling.

Once we have streaming, we can set it up simultaneously for MaRDI and WMF.

[Daniel-Mietchen]

OPTIONAL: set up copies of wikiqlever as needed, along with some load balancer

3. Also, Wikiqlever is currently missing streaming updates, which I would like to integrate before scaling.

Pinging @hannahbast re streaming. This task is now handled separately as

• Set up copies of wikiqlever as needed, along with some load balancer #2771