Skip to content

Improve readme #461

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Phu2 merged 5 commits into from
Feb 19, 2026
Merged

Improve readme #461

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
9e7cc90
Add instructions how to install elasticsearch
TobiasNx Feb 19, 2026
9ae9e74
Add elasticsearch folder to gitignore
TobiasNx Feb 19, 2026
f5e45db
Fix broken link to gnd rdf dumps
TobiasNx Feb 19, 2026
a7e5813
Mark file and code refrences as code
TobiasNx Feb 19, 2026
ddf57a0
Adjust update date format for oai updates
TobiasNx Feb 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,4 @@ bin
*log*.gz
*.tmp
GND-updates*
/elasticsearch-*
46 changes: 30 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,23 @@ Elasticsearch 5.6.x (configured in `application.conf`)

### Build

Get the code, change into the project directory, and run the tests:
Get the code, change into the project directory:

`git clone https://github.com/hbz/lobid-gnd.git ; cd lobid-gnd ; sbt test`
`git clone https://github.com/hbz/lobid-gnd.git ; cd lobid-gnd`

If needed start elasticsearch:

```bash
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.3.tar.gz
tar -xzf elasticsearch-5.6.3.tar.gz
./elasticsearch-5.6.3/bin/elasticsearch &
```

(If elastic search is already running and you want to start a new es: `ps -ef | grep elas` and `kill [PID]`.)

Run tests:

`sbt test`

### Data

Expand Down Expand Up @@ -48,7 +62,7 @@ Index the data, passing the index name:

`sbt -Dindex.entityfacts.index=entityfacts_20210120 "runMain apps.Index entityfacts"`

For configuration details and defaults, see 'conf/application.conf'.
For configuration details and defaults, see `conf/application.conf`.

#### GND Baseline

Expand All @@ -58,13 +72,13 @@ Set up a location for the input data:

`mkdir input_data; cd input_data`

Set 'data.rdfxml' in 'conf/application.conf' to the 'input_data' location.
Set `data.rdfxml` in `conf/application.conf` to the `input_data` location.

Get the GND RDF/XML source data from <https://data.dnb.de/opendata/>:

`wget https://data.dnb.de/opendata/authorities-{geografikum,koerperschaft,kongress,person,sachbegriff,werk}_lds.rdf.gz`
`wget https://data.dnb.de/opendata/authorities-gnd-{geografikum,koerperschaft,kongress,person,sachbegriff,werk}_lds.rdf.gz`

This should give you 6 local files ending with '.rdf.gz'. Go back to the project root directory:
This should give you 6 local files ending with `.rdf.gz`. Go back to the project root directory:

`cd ..`

Expand All @@ -74,11 +88,11 @@ Set up a location for the index data:

`mkdir index_data`

Set 'data.jsonlines' in 'conf/application.conf' to the 'index_data' location.
Set `data.jsonlines` in `conf/application.conf` to the `index_data` location.

Set 'index.boot' in 'conf/application.conf' to an existing index. This index will be used to get labels during the conversion process.
Set `index.boot` in `conf/application.conf` to an existing index. This index will be used to get labels during the conversion process.

Set 'index.prod' in 'conf/application.conf' to a non-existing index. This index name will be used in the indexing data created during conversion.
Set `index.prod` in `conf/application.conf` to a non-existing index. This index name will be used in the indexing data created during conversion.

Convert the data to JSON-LD lines, the index data format:

Expand All @@ -88,11 +102,11 @@ To be able to log out from the server while the conversion is running, we actual

`setsid nohup sbt "runMain apps.ConvertBaseline" &`

This should create 6 '\*.jsonl' files in 'index_data'.
This should create 6 `\*.jsonl` files in `index_data`.

##### Index the JSON data

If the 'index.prod' configured in 'application.conf' does not exists, a new index will be created.
If the `index.prod` configured in `application.conf` does not exists, a new index will be created.

To start the indexing, run:

Expand All @@ -104,11 +118,11 @@ To start the indexing, run:

Updates are pulled via [the DNB OAI-PMH interface](https://www.dnb.de/DE/Professionell/Metadatendienste/Datenbezug/OAI/oai_node.html).

Pass one or two arguments: get updates since (and optionally until) a given date:
Pass one or two arguments: get updates since (and optionally until) a given datetime `[YYYY-MM-DD]T[HH:MM:SS]Z` (not just date):

`sbt "runMain apps.ConvertUpdates 2022-06-22 2022-06-23"`
`sbt "runMain apps.ConvertUpdates 2022-06-22T11:08:23Z 2022-06-23T18:08:23Z"`

The date of the most recent update is stored in 'GND-lastSuccessfulUpdate.txt' (can be changed in the config).
The date of the most recent update is stored in `GND-lastSuccessfulUpdate.txt` (can be changed in the config).

The original downloaded data and the converted data are stored in separate files. To convert the data again without downloading it, use the steps described above under 'Convert RDF/XML to JSON' with the update RDF data.

Expand All @@ -118,11 +132,11 @@ To index the updates run:

`sbt "runMain apps.Index updates"`

See 'application.conf' for details on the configured file names etc.
See `application.conf` for details on the configured file names etc.

### Web

In 'lobid-gnd', run the web application:
In `lobid-gnd`, run the web application:

`sbt run`

Expand Down