Make committed TriG base-relative (base applied at load time)

Committed datasets/current/**/*.trig now use base-relative IRIs with no
@base (e.g. <persons/84854/#this>) instead of absolute
https://linkeddata.lt/… URIs. The base is supplied at load/parse/validate
time, so one committed dataset serves any deployment base (localhost dev,
linkeddata.lt prod) with no regeneration.

RDF stores only absolute IRIs, so the data loaded into Fuseki is identical;
only the git serialization changes. The mappings still mint absolute IRIs
in-model — a new etl/lib/relativize.sh rewrites arq/rdflib output to
relative on write (riot streaming writer + strip @base), leaving external
URIs (Wikidata, Commons, lrs.lt, schema.org, EU tables, mailto/tel) absolute.

- write: relativize.sh; graphify.sh pipes arq through it; reconcile/photos
  become base-aware via rdflib publicID and are relativized on write
- read: validate.sh/shacl.sh/run.sh take a base, resolving relative→absolute
- load: tdb-loader resolves against BASE_URI (.env) via riotcmd.riot before
  tdb2.tdbloader; make load passes -e BASE_URI
- CI (shacl-validation.yml) + docs (CLAUDE.md, README, URI-SCHEME, config.mk)

Verified with Jena 5.6.0: round-trip (riot --base=$B --output=nquads) of
every relativized file is byte-identical to the prior absolute triples;
validation, cross-graph queries, and load-time rebase to both bases pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: namedgraph

.github/workflows/shacl-validation.yml

@@ -18,6 +18,8 @@ on:
 
 env:
   JENA_VERSION: 6.1.0
+  # committed TriG is base-relative; resolve against the prod base for validation
+  BASE: https://linkeddata.lt/
 
 jobs:
   validate:
@@ -58,10 +60,10 @@ jobs:
               # overlay files (owl:sameAs/foaf:depiction onto other files' graphs):
               # graph invariants intentionally absent, so skip validate.sh
               alignments.trig|photos.trig)
-                etl/lib/shacl.sh "etl/shapes/$domain.ttl" "$f" \
+                etl/lib/shacl.sh "etl/shapes/$domain.ttl" "$f" "$BASE" \
                   || { rc=1; echo "::error file=$f::SHACL validation failed"; echo "- FAIL: $f" >> "$GITHUB_STEP_SUMMARY"; } ;;
               *)
-                etl/lib/validate.sh "$f" \
+                etl/lib/validate.sh "$f" "$BASE" \
                   || { rc=1; echo "::error file=$f::validation failed"; echo "- FAIL: $f" >> "$GITHUB_STEP_SUMMARY"; } ;;
             esac
             echo "::endgroup::"

CLAUDE.md

@@ -10,12 +10,13 @@ cd etl && make                 # everything: taxonomies → admin-units → seim
 make -C etl/<domain> all       # one domain (fetch is always fresh — FORCE prerequisite)
 make -C etl/seimas photos      # opt-in: scrape official portraits from lrs.lt
 make BASE=https://linkeddata.lt/   # prod base URI (default https://localhost:4443/, see etl/config.mk);
-                               # committed datasets/current/ are generated with the prod base
+                               # committed datasets/current/ are base-RELATIVE (no @base) — identical
+                               # for any BASE; the base is (re)applied at load/parse/validate time
 
 etl/queries/run.sh <q.rq>      # SPARQL over ALL datasets loaded in-memory (~1M quads, -Xmx4g)
 python3 etl/queries/render-examples.py   # regenerate etl/queries/EXAMPLES.md result tables
 
-uv run --project etl/tools ltlod-reconcile <admin-units|persons> --input … --output …
+uv run --project etl/tools ltlod-reconcile <admin-units|persons> [--base …] --input … --output …
 
 make sef                       # compile the client-side XSLT override (files/client.xsl) to a
                                # Saxon-JS SEF (files/client.xsl.sef.json, gitignored). Run once
@@ -33,8 +34,8 @@ make install                   # set up the dataspace via LDH CLI: make it publi
                                # or `printf '\n\n\n\n' | make install`); enter another Base URL
                                # + owner cert to install onto any LDH instance
 make load                      # bulk-load datasets/current/*/*.trig into fuseki-end-user TDB2;
-                               # regenerate with `make -C etl` first (committed data has prod base);
-                               # ends with `make public` (anonymous read, LDH make-public.sh equivalent)
+                               # resolves the base-relative TriG against BASE_URI (.env) via riot
+                               # before tdb2.tdbloader; ends with `make public` (anonymous read)
 make down / make drop          # stop stack / wipe LDH runtime state (never datasets/current/)
 ```
 
@@ -53,13 +54,17 @@ Every domain runs the same four stages (shared scripts in `etl/lib/`):
 2. **normalize** — CSV → CSV2RDF identity transform (docker), XML → XSLT 1.0
    (`xsltproc`), both emit source-shaped RDF with `<{base}#column>` properties.
 3. **graphify** — LDH-style quad `CONSTRUCT { GRAPH ?graph {…} }` mapping
-   (`mappings/*.rq`, `$base`-parameterized) executed by `arq` → TriG. The `.rq`
-   files are reusable verbatim as LinkedDataHub CSV imports.
+   (`mappings/*.rq`, `$base`-parameterized) executed by `arq`; the CONSTRUCT mints
+   ABSOLUTE IRIs in-model, then `etl/lib/relativize.sh` rewrites the output to
+   base-RELATIVE IRIs with no `@base` → TriG. The `.rq` files are reusable verbatim
+   as LinkedDataHub CSV imports. (`ltlod-reconcile`/photos take `--base` as the
+   rdflib publicID and are relativized the same way; see the relative-TriG gotcha.)
 4. **validate** — `riot --validate` + every graph must have `dct:title` and
    `foaf:primaryTopic` on the graph URI (see `etl/lib/validate.sh`) + SHACL
    shapes per entity type (`etl/shapes/<domain>.ttl`, auto-selected by output
    dir, executed via `etl/lib/shacl.sh`; also run in CI on committed datasets
-   by `.github/workflows/shacl-validation.yml`).
+   by `.github/workflows/shacl-validation.yml`). Both `validate.sh` and `shacl.sh`
+   take a base to resolve the relative IRIs (validate.sh pre-resolves to N-Quads).
 
 Post-ETL: `ltlod-reconcile` matches entities to Wikidata (closed candidate sets
 via WDQS, exact label + parent disambiguation) and writes `owl:sameAs` + images
@@ -166,9 +171,21 @@ merge on load). Unmatched entities go to `cache/unmatched*.csv`, never force-mat
 - **`make load` bypasses LDH's HTTP API**: it runs `tdb2.tdbloader` directly against
   the end-user TDB2 store via the one-off `tdb-loader` compose service (the
   `atomgraph/fuseki` image bundles the full Jena CLI inside the fuseki-server jar).
+  Because the committed TriG is base-relative, the service first resolves it against
+  `BASE_URI` (from `.env`, passed as `-e BASE_URI=…`) with `riotcmd.riot --base=…
+  --output=nquads`, then loads the N-Quads (tdb2.tdbloader has no `--base`).
   Load is append-only — clean rebuild: `make down && rm -rf fuseki/end-user &&
   make up && make load`. It stops fuseki-end-user first and removes the stale
   `tdb.lock` (lock PIDs are container-relative), then restarts the Varnish caches.
+- **Committed TriG is base-RELATIVE with no `@base`** (base-agnostic: one dataset
+  serves any deployment base). RDF stores only absolute IRIs, so the base must be
+  reapplied on EVERY read — always pass `--base` to `riot`/`arq`/`shacl`
+  (`riot --base=$BASE …`) or relative IRIs resolve to `file://…` garbage. The write
+  path (`etl/lib/relativize.sh`) uses riot's STREAMING trig writer (`--output`, which
+  relativizes against a prepended `@base`; `--formatted` does NOT) and strips the base
+  header. Only base-internal IRIs relativize; external URIs (Wikidata, Commons, lrs.lt,
+  EU tables, schema.org, mailto/tel) stay absolute. Round-trip proof:
+  `riot --base=$BASE --output=nquads file.trig | sort` is base-independent.
 - **Fuseki ports are never published to the host** — query via
   `https://localhost:4443/sparql` or from inside the network:
   `docker compose exec linkeddatahub curl http://varnish-end-user/ds/`.

Makefile

@@ -106,8 +106,10 @@ install:
 		./app/install.sh "$$BASE_URL" "$$CERT_PATH" "$$PASSWORD"; \
 	fi
 
-# Bulk-load datasets/current/*/*.trig into the end-user TDB2 store. APPEND-ONLY:
-# clean rebuild = `make down && rm -rf fuseki/end-user && make up && make load`.
+# Bulk-load datasets/current/*/*.trig into the end-user TDB2 store. The committed
+# TriG is base-relative; the loader resolves it against BASE_URI (from .env) so
+# the same files load at whatever base this deployment uses — no per-base regen.
+# APPEND-ONLY: clean rebuild = `make down && rm -rf fuseki/end-user && make up && make load`.
 load:
 	@ls datasets/current/*/*.trig >/dev/null 2>&1 || \
 		{ echo "ERROR: no TriG files under datasets/current/ — run 'make -C etl' first."; exit 1; }
@@ -119,7 +121,7 @@ load:
 	done
 	docker compose stop fuseki-end-user
 	rm -f fuseki/end-user/DB2/tdb.lock
-	docker compose run --rm tdb-loader
+	docker compose run --rm -e BASE_URI="$(BASE_URI)" tdb-loader
 	docker compose up -d fuseki-end-user
 	docker compose restart varnish-end-user varnish-frontend
 	$(MAKE) public

README.md

@@ -38,7 +38,9 @@ todėl grafas saugo ne tik dabartinę būseną, bet ir **istoriją**: pasikeitus
 narystė gauna pabaigos datą, o nauja — pradžios.
 
 Kiekvienas objektas gyvena savo **named graph'e** (dokumente), kurio URI sutampa su dokumento
-adresu. Štai visas Birštono savivaldybės dokumentas ([TriG](https://www.w3.org/TR/trig/) sintakse):
+adresu. Štai visas Birštono savivaldybės dokumentas ([TriG](https://www.w3.org/TR/trig/) sintakse;
+čia parodyta išskleista absoliuti forma — užfiksuoti failai naudoja santykines URI, pvz.
+`<admin-units/12/#this>`, o bazė pritaikoma krovimo metu):
 
 ```turtle
 <https://linkeddata.lt/admin-units/12/> {
@@ -153,18 +155,20 @@ make up      # sugeneruoja slaptažodžius + serverio sertifikatą ir paleidžia
 make install # suteikia viešą skaitymo prieigą, sukuria konteinerių dokumentus ir vardų erdvės
              # ontologiją per LDH CLI (reikia ../LinkedDataHub);
              # interaktyvus: Enter×4 = lokali aplinka, kitas Base URL = bet kuri LDH instancija
-make -C etl  # perkuria rinkinius su numatytąja baze https://localhost:4443/
 make load    # užkrauna datasets/current/*/*.trig tiesiai į triplestore
+             # (nereikia perkurti — rinkiniai bazei neutralūs; bazė pritaikoma krovimo metu)
 ```
 
 Po `make up` LDH pasiekiamas adresu **<https://localhost:4443/>** (savo pasirašytas
 sertifikatas — naršyklė įspės; pirmas paleidimas trunka ~1–2 min.). Administravimo
 aplinka — <https://admin.localhost:4443/>.
 
 **Svarbu dėl bazinės URI:** repozitorijoje užfiksuoti `datasets/current/` failai
-sugeneruoti su produkcine baze `https://linkeddata.lt/`, tad prieš `make load` rinkinius
-reikia perkurti su numatytąja lokalia baze (`make -C etl`) — kitaip dokumentų URI
-nesutaps su LDH adresu ir jie nebus pasiekiami.
+naudoja **santykines URI** (be `@base`), tad yra neutralūs bazei — tas pats rinkinys
+tinka bet kuriam diegimui. `make load` išsprendžia santykines URI pagal `.env` bazę
+(`BASE_URI`, lokaliai `https://localhost:4443/`) su `riot` prieš `tdb2.tdbloader`, tad
+dokumentų URI visada sutampa su LDH adresu. Perkurti prieš krovimą nebūtina —
+`make -C etl` reikia tik norint atsinaujinti duomenis iš gyvų šaltinių.
 
 Duomenų struktūra kuriama dviem lygiais:
 
@@ -180,8 +184,9 @@ Duomenų struktūra kuriama dviem lygiais:
   repozitorijos (`../LinkedDataHub`, keičiama per `make install LDH_HOME=…`).
 - **Duomenys** (`make load`): ETL rinkiniai — vien `dh:Item` dokumentai su
   `sioc:has_container` nuorodomis į karkasą — rašomi **tiesiogiai į
-  `fuseki-end-user` TDB2 saugyklą** (`tdb2.tdbloader` per vienkartinį
-  `tdb-loader` konteinerį), ne po vieną dokumentą per HTTP: ~1 mln. ketvertų
+  `fuseki-end-user` TDB2 saugyklą** (santykinės URI pirma išsprendžiamos pagal
+  `.env` bazę su `riot`, tada `tdb2.tdbloader` per vienkartinį `tdb-loader`
+  konteinerį), ne po vieną dokumentą per HTTP: ~1 mln. ketvertų
   užsikrauna per kelias minutes. Pabaigoje suteikiama vieša skaitymo prieiga
   (`make public` — LDH CLI `make-public.sh` atitikmuo, vykdomas tiesiogiai per
   `fuseki-admin` konteinerių tinkle).
@@ -275,8 +280,9 @@ Daugiau klausimų, į kuriuos duomenys jau atsako (visi su užklausomis ir pilna
 
 **Publikavimas.** Lokalus LinkedDataHub diegimas jau yra — žr.
 [Publikavimas su LinkedDataHub](#publikavimas-su-linkeddatahub). Lieka produkcinis
-diegimas `https://linkeddata.lt/` adresu (rinkiniai jau generuojami su `$base`
-parametrizuotomis URI, tad tereikia `make -C etl BASE=https://linkeddata.lt/`).
+diegimas `https://linkeddata.lt/` adresu (rinkiniai bazei neutralūs — santykinės URI —
+tad tuos pačius `datasets/current/` failus užtenka užkrauti su produkcine `.env` baze;
+perkurti nereikia).
 
 **Nauji rinkiniai** (integracijos taškai jau paruošti — žr. „kaip pridėti“ žemiau):