Get your own copy of scholia - example installation #2760

WolfgangFahl · 2025-12-15T08:17:47Z

WolfgangFahl
Dec 15, 2025

This is a script of a concrete example installation see #2759 for the general approach. See #2756 for the motivation.

Wikidata Query service

get data.jnl seed file

see https://wiki.bitplan.com/index.php/Wikidata_Import_2025-12-13

./jnlget --cat
✓ md5sums.txt exists
➜ reassembling downloads into 2025-12-07-wikidata-data.jnl.gz
⚠ 2025-12-07-wikidata-data.jnl.gz exists will not override
✓ reassembled 2025-12-07-wikidata-data.jnl.gz (438103694405 bytes)
➜ checking md5 for 2025-12-07-wikidata-data.jnl.gz of size 438103694405 to be ad0006a38103efd715c782a539a6f482

launch

./launch

configure proxy

cat /etc/apache2/sites-available/wdqs.conf 
# Apache Configuration for wdqs.bitplan.com  
# WDQS Frontend - Port 8099
<VirtualHost *:80>
    ServerName wdqs.bitplan.com
    ServerAdmin [email protected]
    
    ErrorLog ${APACHE_LOG_DIR}/wdqs.log
    CustomLog ${APACHE_LOG_DIR}/wdqs.log combined
    ProxyPass / http://localhost:8099/
    ProxyPassReverse / http://localhost:8099/
</VirtualHost>

test

# show the number of triples and the timestamp of the last modification
PREFIX schema: <http://schema.org/>

SELECT
  (?count as ?tripleCount)
  ?dateModified
  (STR(?dateModified) as ?timestamp)
WHERE {
  {
    SELECT (COUNT(*) AS ?count) {
      ?s ?p ?o
    }
  }
  OPTIONAL {
    <http://www.wikidata.org> schema:dateModified ?dateModified
  }
}

screenshot

QLever

QLever control script

pipx install qlever --force
Installing to existing venv 'qlever'
  installed package qlever 0.5.35, installed using Python 3.10.12
  These apps are now globally available
    - qlever
done! ✨ 🌟 ✨

System info

qlever system-info
Version: 0.5.35 (qlever --version)
OS: Linux (Ubuntu 22.04.5 LTS)
Arch: x86_64
Host: fur
RAM: 125.7 GB total, 113.1 GB available
CPU: 12 Cores, 24 Threads @ 3.70 GHz

configure wikidata

This needs to be on a 2 TB SSD space to be efficient

qlever setup-config wikidata

To enable autocompletion, run the following command, and consider adding it to your `.bashrc` or `.zshrc`:

eval "$(register-python-argcomplete qlever)" && export QLEVER_ARGCOMPLETE_ENABLED=1


Command: setup-config

cat /home/wf/.local/pipx/venvs/qlever/lib/python3.10/site-packages/qlever/Qleverfiles/Qleverfile.wikidata | sed -E 's/(^ACCESS_TOKEN.*)/\1_fMZMyAWy6pp1/'> Qleverfile

Created Qleverfile for config "wikidata" in current directory

get data

 qlever get-data

To enable autocompletion, run the following command, and consider adding it to your `.bashrc` or `.zshrc`:

eval "$(register-python-argcomplete qlever)" && export QLEVER_ARGCOMPLETE_ENABLED=1


Command: get-data

curl -LRC - -O https://dumps.wikimedia.org/wikidatawiki/entities/latest-all.ttl.bz2 -O https://dumps.wikimedia.org/wikidatawiki/entities/latest-lexemes.ttl.bz2 2>&1 | tee wikidata.download-log.txt && curl -sL https://dumps.wikimedia.org/wikidatawiki/entities/dcatap.rdf | docker run -i --rm -v $(pwd):/data stain/jena riot --syntax=RDF/XML --output=NT /dev/stdin > dcatap.nt

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   153  100   153    0     0    449      0 --:--:-- --:--:-- --:--:--   450
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   153  100   153    0     0   1813      0 --:--:-- --:--:-- --:--:--  1821
Download successful, total file size: 110,987 bytes

wf@fur:/hd/tepig/qlever/wikidata_20251215$ ls -l
total 128
-rw-rw-r-- 1 wf wf 110681 Dez 15 09:23 dcatap.nt
-rw-rw-r-- 1 wf wf    153 Dez 15 09:23 latest-all.ttl.bz2
-rw-rw-r-- 1 wf wf    153 Dez 15 09:23 latest-lexemes.ttl.bz2
-rw-rw-r-- 1 wf wf   2074 Dez 15 09:19 Qleverfile
-rw-rw-r-- 1 wf wf    634 Dez 15 09:23 wikidata.download-log.txt
wf@fur:/hd/tepig/qlever/wikidata_20251215$ cat latest-all.ttl.bz2 
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0</center>
</body>
</html>

see qlever-dev/qlever-control#223

we need to work around and get the data from a working environment

ssh wikidata.dbis.rwth-aachen.de ls -l /hd/delta/qlever/wikidata_20251210/*.bz2
-rw-rw-r-- 1 wf wf 120810121675 Dec  4 14:45 /hd/delta/qlever/wikidata_20251210/latest-all.ttl.bz2
-rw-rw-r-- 1 wf wf    601425580 Dec  6 00:37 /hd/delta/qlever/wikidata_20251210/latest-lexemes.ttl.bz2
nohup scp -p wikidata.dbis.rwth-aachen.de:/hd/delta/qlever/wikidata_20251210/*.bz2 .&

index

nohup qlever index&

start

qlever start

To enable autocompletion, run the following command, and consider adding it to your `.bashrc` or `.zshrc`:

eval "$(register-python-argcomplete qlever)" && export QLEVER_ARGCOMPLETE_ENABLED=1


Command: start

docker run -d --restart=unless-stopped -u $(id -u):$(id -g) -v /etc/localtime:/etc/localtime:ro -v $(pwd):/index -p 7001:7001 -w /index --name qlever.server.wikidata --init --entrypoint bash adfreiburg/qlever -c 'ServerMain -i wikidata -j 8 -p 7001 -m 20G -c 15G -e 5G -k 200 -s 600s -a wikidata_fMZMyAWy6pp1 > wikidata.server-log.txt 2>&1'

Follow wikidata.server-log.txt until the server is ready (Ctrl-C stops following the log, but NOT the server)

2025-12-15 20:48:02.407 - INFO: QLever Server, compiled on Sun Dec 14 23:56:56 UTC 2025 using git hash a572da
2025-12-15 20:48:02.408 - INFO: Initializing server ...
2025-12-15 20:48:02.408 - INFO: The git hash used to build this index was "a8c9d8"
2025-12-15 20:48:02.408 - INFO: Reading vocabulary from file wikidata.vocabulary.words ...
2025-12-15 20:48:02.724 - INFO: Done, number of words: 3,752,600,327
2025-12-15 20:48:02.732 - INFO: Number of words in internal vocabulary (these are also part of the external vocabulary): 3,780,604
2025-12-15 20:48:02.944 - INFO: Registered PSO permutation: #relations = 25,663, #blocks = 297,513, #triples = 9,221,705,501
2025-12-15 20:48:03.421 - INFO: Registered PSO permutation: #relations = 61,652, #blocks = 674,891, #triples = 20,953,219,214
2025-12-15 20:48:03.610 - INFO: Registered POS permutation: #relations = 25,663, #blocks = 297,513, #triples = 9,221,705,501
2025-12-15 20:48:04.085 - INFO: Registered POS permutation: #relations = 61,652, #blocks = 674,891, #triples = 20,953,219,214
2025-12-15 20:48:04.500 - INFO: Registered OPS permutation: #relations = 3,794,947,505, #blocks = 586,258, #triples = 20,953,219,214
2025-12-15 20:48:04.914 - INFO: Registered OSP permutation: #relations = 3,794,947,505, #blocks = 586,258, #triples = 20,953,219,214
2025-12-15 20:48:05.193 - INFO: Registered SPO permutation: #relations = 2,304,083,297, #blocks = 447,463, #triples = 20,953,219,214
2025-12-15 20:48:05.460 - INFO: Registered SOP permutation: #relations = 2,304,083,297, #blocks = 447,463, #triples = 20,953,219,214
2025-12-15 20:48:05.460 - INFO: Reading patterns from file wikidata.index.patterns ...
2025-12-15 20:48:14.337 - INFO: Sorting random result tables to estimate the sorting performance of this machine ...
2025-12-15 20:48:17.307 - INFO: Access token for restricted API calls is "wikidata_fMZMyAWy6pp1"
2025-12-15 20:48:17.307 - INFO: The server is ready, listening for requests on port 7001 ...
2025-12-15 20:48:17.561 - INFO: 
2025-12-15 20:48:17.561 - INFO: Request received via POST, content type "application/x-www-form-urlencoded"
2025-12-15 20:48:17.561 - INFO: Alive check with message "from the `qlever` CLI"
2025-12-15 20:48:17.569 - INFO: 
2025-12-15 20:48:17.569 - INFO: Request received via GET, no content type specified
2025-12-15 20:48:17.569 - INFO: Setting index description to: "Complete Wikidata, from https://dumps.wikimedia.org/wikidatawiki/entities (latest-all.ttl.bz2 and latest-lexemes.ttl.bz2), version 04.12.2025"

curl -s fur:7001 --data-urlencode "cmd=cache-stats"
curl -s fur:7001 --data-urlencode "cmd=get-settings"

Cache is empty, all 15.0 GB available

ui

we need a Qleverfile-ui.yml with a proper

config:
   backend:
      name: wikidata
      slug: wikidata
      baseUrl: https://qlever-api.bitplan.com

qlever ui

qlever apache configs

qlever for frontend/ui

cat qlever.conf 
# Apache Configuration for qlever.bitplan.com
# # Generated by omnigraph at 2025-11-01T13:01:19.943163
# omnigraph Version 0.1.1 of 2025-06-13 (Unified Python interface for multiple graph databases)
# http Port: 8176
# SSL Port: 443
# timeout: 5400
<VirtualHost *:443>
    ServerName qlever.bitplan.com
    ServerAdmin [email protected]

    Include ssl.conf
    ErrorLog ${APACHE_LOG_DIR}/qlever_error_ssl.log
    CustomLog ${APACHE_LOG_DIR}/qlever_ssl.log combined

    ProxyPreserveHost On
    ProxyTimeout 5400

    ProxyPass / http://localhost:8176/
    ProxyPassReverse / http://localhost:8176/
</VirtualHost>
<VirtualHost *:80>
    ServerName qlever.bitplan.com
    ServerAdmin [email protected]

    
    ErrorLog ${APACHE_LOG_DIR}/qlever_error.log
    CustomLog ${APACHE_LOG_DIR}/qlever.log combined

    ProxyPreserveHost On
    ProxyTimeout 5400

    ProxyPass / http://localhost:8176/
    ProxyPassReverse / http://localhost:8176/
</VirtualHost>

qlever-api for backend

# /etc/apache2/sites-available/qlever-api.conf 
# QLever API Wikidata HTTPS configuration
# Last updated: 2025-10-29
<VirtualHost *:443>
  ServerName qlever-api.bitplan.com 
  ServerAdmin webmaster@localhost
  ErrorLog ${APACHE_LOG_DIR}/qlever-api_error.log
  CustomLog ${APACHE_LOG_DIR}/qlever-api.access.log combined
  
  ProxyPreserveHost On
  Timeout 5400
  ProxyTimeout 5400
  
  # Remove duplicate CORS headers from backend, set once
  Header unset Access-Control-Allow-Origin
  Header always set Access-Control-Allow-Origin "*"
  Header always set Access-Control-Allow-Methods "GET, POST, OPTIONS"
  Header always set Access-Control-Allow-Headers "Content-Type, Accept"
  
  ProxyPass / http://localhost:7001/
  ProxyPassReverse / http://localhost:7001/
  
  Include ssl.conf
</VirtualHost>

<VirtualHost *:80>
  ServerName qlever-api.bitplan.com 
  RewriteEngine On
  RewriteRule ^(.*)$ https://%{HTTP_HOST}$1 [R=301,L]
</VirtualHost>

Scholia

see https://cr.bitplan.com/index.php/Scholia

clone

git clone https://github.com/WDscholia/scholia
Cloning into 'scholia'...
remote: Enumerating objects: 20921, done.
remote: Counting objects: 100% (2722/2722), done.
remote: Compressing objects: 100% (613/613), done.
remote: Total 20921 (delta 2328), reused 2109 (delta 2109), pack-reused 18199 (from 2)
Receiving objects: 100% (20921/20921), 5.26 MiB | 4.07 MiB/s, done.
Resolving deltas: 100% (15785/15785), done.

install

pipx install .
  installed package scholia 0.3+1196.gb14be14d, installed using Python 3.10.12
  These apps are now globally available
    - scholia
done! ✨ 🌟 ✨

usage

scholia -version
Usage:
  scholia arxiv-to-quickstatements [options] <arxiv>
  scholia orcid-to-q <orcid>
  scholia string-to-type <string>
  scholia run

apache config

# /etc/apache2/sites-available/scholia.conf
# Apache Configuration for scholia.bitplan.com
# intranet version
# scholia Version 0.3 of 2025-12-12
# http Port: 8100
# SSL Port: 443

<VirtualHost *:443>
    ServerName scholia.bitplan.com
    ServerAdmin [email protected]

    # Your shared SSL settings and certificates
    Include ssl.conf

    # Reverse proxy to the app on 8100
    ProxyPreserveHost On
    RequestHeader set X-Forwarded-Proto "https"
    ProxyPass / http://127.0.0.1:8100/
    ProxyPassReverse / http://127.0.0.1:8100/

    ErrorLog ${APACHE_LOG_DIR}/scholia_error_ssl.log
    CustomLog ${APACHE_LOG_DIR}/scholia_ssl.log combined
</VirtualHost>

<VirtualHost *:80>
    ServerName scholia.bitplan.com
    ServerAdmin [email protected]

    Redirect permanent / https://scholia.bitplan.com/

    ErrorLog ${APACHE_LOG_DIR}/scholia_error.log
    CustomLog ${APACHE_LOG_DIR}/scholia.log combined
</VirtualHost>

enable and start

sudo a2ensite scholia
sudo service apache2 restart

Qlever branch

clone

git clone https://github.com/ad-freiburg/scholia.git scholia-qlever
Cloning into 'scholia-qlever'...
remote: Enumerating objects: 21844, done.
remote: Counting objects: 100% (2879/2879), done.
remote: Compressing objects: 100% (560/560), done.
remote: Total 21844 (delta 2512), reused 2344 (delta 2319), pack-reused 18965 (from 3)
Receiving objects: 100% (21844/21844), 5.56 MiB | 3.19 MiB/s, done.
Resolving deltas: 100% (16650/16650), done.
wf@fur:~/source/python$ cd scholia-qlever/
wf@fur:~/source/python/scholia-qlever$ git checkout qlever
Branch 'qlever' set up to track remote branch 'qlever' from 'origin'.
Switched to a new branch 'qlever'

apache

see above but with scholia-qlever on port 8101

WolfgangFahl · 2025-12-15T13:13:12Z

WolfgangFahl
Dec 15, 2025
Author

startup script see #2761

0 replies

WolfgangFahl · 2025-12-15T20:17:14Z

WolfgangFahl
Dec 15, 2025
Author

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Get your own copy of scholia - example installation #2760

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Select a reply

Uh oh!

Get your own copy of scholia - example installation #2760

Uh oh!

Uh oh!

WolfgangFahl Dec 15, 2025

Wikidata Query service

get data.jnl seed file

launch

configure proxy

test

screenshot

QLever

QLever control script

System info

configure wikidata

get data

index

start

ui

qlever apache configs

qlever for frontend/ui

qlever-api for backend

Scholia

clone

install

usage

apache config

enable and start

Qlever branch

clone

apache

Replies: 2 comments

Uh oh!

WolfgangFahl Dec 15, 2025 Author

Uh oh!

Uh oh!

WolfgangFahl Dec 15, 2025 Author

WolfgangFahl
Dec 15, 2025

WolfgangFahl
Dec 15, 2025
Author

WolfgangFahl
Dec 15, 2025
Author