Skip to content

README.md

Latest commit

 

History

History
109 lines (67 loc) · 9.08 KB

README.md

File metadata and controls

109 lines (67 loc) · 9.08 KB
Raw

Wikibase Suite Wikidata Query Service (wdqs) image

The Wikidata Query Service (WDQS) provides a way for tools to access Wikibase data, via a SPARQL API. It is based on Blazegraph.

💡 This image is part of Wikibase Suite (WBS) which provides everything you need to run a Wikibase instance on your own server.

Requirements

In order to run WDQS, you need:

  • at least 2 GB RAM to start WDQS
  • MediaWiki/Wikibase instance
  • WDQS as server
  • WDQS as updater
  • WDQS Proxy for public facing setups
  • Configuration via environment variables

MediaWiki/Wikibase instance

We suggest using the WBS Wikibase image because this is the image we run all our tests against. Follow the setup instructions over there to get it up and running.

WDQS as server

You'll need one instance of the image to execute the actual WDQS daemon started using /runBlazegraph.sh.

You can send GET requests with your SPARQL query to the WDQS endpoint (following the example below): http://wdqs:9999/bigdata/namespace/wdq/sparql?query={SPARQL}

WDQS as updater

You'll need one instance of the image to execute the updater started using /runUpdate.sh. This polls changes from Wikibase.

Reverse proxy

By default, WDQS exposes some endpoints and methods that reveal internal details or functionality that might not be intended in every setup, especially when running as a public service. The example below includes a traefik proxy configuration limiting the functionality WDQS exposes.

Environment variables

Variables in bold are required.

Variable Default Description
WIKIBASE_HOST "wikibase" Hostname to reach the Wikibase service, e.g. the docker network internal hostname
WIKIBASE_CONCEPT_URI "" Concept URI, required for /runUpdate.sh only, the identifying prefix to entities in this knowledge graph, e.g. the public URL of the Wikibase host.
WDQS_HOST "wdqs" WDQS hostname (this service)
WDQS_PORT "9999" WDQS port (this service)
WIKIBASE_SCHEME "http" URL scheme used to reach the Wikibase service, e.g. http to reach a local wikibase on the same docker network
WDQS_ENTITY_NAMESPACES "120,122" Wikibase namespaces to load data from
WIKIBASE_MAX_DAYS_BACK "90" Maximum number of days updater can reach back in time from now
MEMORY "" Memory limit for Blazegraph
HEAP_SIZE "1g" Heap size for Blazegraph
BLAZEGRAPH_EXTRA_OPTS "" Extra options to be passed to Blazegraph,they must be prefixed with -D. Example: -Dhttps.proxyHost=http://my.proxy.com -Dhttps.proxyPort=3128. See the WDQS User Manual.

Example

For an integrated Docker Compose example showing how this image is used in the full Wikibase Suite configuration, see deploy/docker-compose.yml.

Releases

Official releases of this image can be found on Docker Hub wikibase/wdqs.

Versioning

This image uses the shared WBS image tag format. See Wikibase Suite image versioning.

In addition to the standard tags, this image also publishes a tag that includes the bundled WDQS version.

Tag Example Description
MAJOR.MINOR.PATCH_wdqsWDQS-VERSION 3.1.7_wdqs0.1.317 Same as the standard patch-version tag, but also mentions the bundled WDQS version.

Upgrading

When upgrading between WDQS versions, the data stored in /wdqs/data may not be compatible with the newer version. When testing the new image, if no data appears to have been loaded into the Query Service, you'll need to reload the data.

If all changes still appear in [RecentChanges], removing /wdqs/data and restarting the service should reload all data.

However, [RecentChanges] are periodically purged of older entries, as determined by the MediaWiki configuration $wgRCMaxAge.

If you can't use [RecentChanges], you'll need to reload from an RDF dump:

Internal filesystem layout

Hooking into the internal filesystem can extend the functionality of this image.

File Description
/wdqs/allowlist.txt SPARQL endpoints allowed for federation
/wdqs/RWStore.properties Properties for the service
/templates/mwservices.json Template for MediaWiki services (populated and placed into /wdqs/mwservices.json at runtime)

Empty-store updater initialization

Before starting, /runUpdate.sh checks whether WDQS contains any entities. When WDQS is empty, it initializes the updater checkpoint from the beginning of the UTC day containing the oldest available RecentChanges entry in the configured Wikibase entity namespaces. This allows a freshly installed or previously affected instance to leave a restart loop and import all entities still represented in RecentChanges. If there are no retained RecentChanges entries, it initializes the checkpoint at the beginning of the current UTC day.

The initialization is guarded: if WDQS contains an entity, or if either service check fails or returns an unexpected response, the updater starts normally without changing its checkpoint. Entities no longer represented in RecentChanges require a full reload as described in Upgrading.

Source

This image is built from this Dockerfile.

Authors & contact

This image is maintained by the Wikibase Suite Team at Wikimedia Germany (WMDE).

If you have questions not listed above or need help, use this bug report form to start a conversation with the engineering team.