Skip to content

Commit 6ebeb83

Browse files
turbomamclaude
andauthored
Add SPARQL QC meta-test guardrail (closes #500) (#505)
* Add SPARQL QC meta-test guardrail (#500) The QC violation queries filter terms on the METPO namespace. That filter was once 'https://w3id.org/METPO_' instead of 'https://w3id.org/metpo/', matching nothing, so every check reported '0 violations' and CI stayed green (#487). A check that cannot fail is false confidence. Add a positive-control fixture (one seeded violation per query) and a runner that asserts each *-violation.sparql returns >= 1 row against it. Queries are discovered dynamically, so a new violation query with no fixture case fails here. Verified locally: passes on the real queries, and fails (exit 1) when a query's namespace filter is reverted to the broken 'METPO_' form. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Document the QC SPARQL meta-test contract (#500) Codify the standing rule so it is written down, not only enforced: every *-violation.sparql needs a seeded case in qc-positive-fixture.ttl, and the meta-test fails any query that matches zero rows. Add src/sparql/tests/README.md (co-located with the fixture/runner) and a pointer in CLAUDE.md's validation section. Purely additive: no files deleted, no wiring changed. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Move the SPARQL QC meta-test out of the ODK-managed tree (#500) Per policy, nothing of ours goes in ODK-owned paths (src/ontology, src/sparql). Relocate the meta-test fixture, runner, and README from src/sparql/tests/ to tests/sparql_qc/ (the non-ODK test tree that CI already runs). The runner now reads the violation queries from src/sparql/ read-only; it writes nothing there. Update the workflow path and the CLAUDE.md pointer. Verified in the container: all 5 queries still match their fixture cases. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Clarify fixture header: one seeded case per query, >= 1 row (#505) Per Copilot review: a query may emit more than one row for a single seeded case (multiple-replaced_by joins the value pair both ways), so the header no longer implies a strict 1:1; the meta-test asserts >= 1 row. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Harden sparql-metatest workflow: pin checkout, least-privilege, persist-credentials Match the workflow-hardening pattern (#507) so this new workflow is consistent and passes zizmor once that lands: pin actions/checkout to a commit SHA, add 'permissions: contents: read', and set persist-credentials: false. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Ontology-header metadata QC, kept outside the ODK tree (closes #502) (#506) * Header-metadata QC, kept outside the ODK tree (#502) Per policy, do not edit ODK-owned queries. Instead of extending src/sparql/dc-properties-violation.sparql, the header check is its own query in tests/sparql_qc/ontology-header-check.sparql, run by the meta-test runner: it must match >= 1 row against the bad-header fixture node (proven able to fail) and 0 rows against the real metpo.owl. Flags creator/contributor given as a literal placeholder instead of an agent IRI, and dcterms:modified disagreeing with owl:versionInfo. The deprecated DC Elements 1.1 namespace stays covered by ODK's untouched dc-properties-violation. Verified in the container: clean repo passes; a dirtied header fails with a clear message; metpo.owl unchanged. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Scope the header check to the METPO ontology IRI; catch non-IRI agents (#506) Per Copilot review: restrict the ontology-header check to the METPO ontology node (https://w3id.org/metpo/) so a merged/imported ontology header cannot raise a false positive, and flag any non-IRI creator/contributor (blank nodes too), not only literals. Verified: clean metpo.owl = 0, fixture = 2, dirty header caught. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent abb90da commit 6ebeb83

6 files changed

Lines changed: 315 additions & 0 deletions

File tree

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
name: SPARQL QC meta-test
2+
3+
# Guardrail for berkeleybop/metpo#500: prove the QC SPARQL violation queries
4+
# actually match METPO terms, so they cannot silently become no-ops.
5+
#
6+
# The queries in src/sparql/*-violation.sparql filter terms with
7+
# STRSTARTS(str(?term), "https://w3id.org/metpo/")
8+
# That filter was once "https://w3id.org/METPO_" (matches nothing), so every
9+
# check reported "0 violations" for a long time and CI stayed green (#487,
10+
# #465). This job runs each query against a positive-control fixture seeded
11+
# with one known violation per query and fails if any matches 0 rows.
12+
13+
on:
14+
push:
15+
branches: [main]
16+
pull_request:
17+
branches: [main]
18+
workflow_dispatch:
19+
20+
permissions:
21+
contents: read
22+
23+
jobs:
24+
sparql-metatest:
25+
runs-on: ubuntu-latest
26+
container: obolibrary/odkfull:v1.6.1
27+
steps:
28+
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
29+
with:
30+
persist-credentials: false
31+
32+
- name: Assert each QC violation query matches its positive-control fixture
33+
shell: bash
34+
run: bash tests/sparql_qc/run_sparql_metatest.sh

CLAUDE.md

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,16 @@ cd src/ontology
155155
sh run.sh make reason_test sparql_test
156156
```
157157

158+
> **QC SPARQL checks must be provable.** The `sparql_test` violation queries in
159+
> `src/sparql/*-violation.sparql` filter on the METPO namespace; a wrong filter
160+
> once matched nothing and reported "0 violations" while CI stayed green (#487).
161+
> Every `*-violation.sparql` therefore needs a seeded case in
162+
> `tests/sparql_qc/qc-positive-fixture.ttl`; the meta-test
163+
> (`.github/workflows/sparql-metatest.yml`) fails if any query matches zero rows.
164+
> When you add or edit a check, add/adjust its fixture case in the same PR. The
165+
> harness lives under `tests/` (not the ODK-managed `src/` tree); see
166+
> `tests/sparql_qc/README.md`.
167+
158168
### GitHub Actions workflow tooling
159169

160170
The CI workflows are linted and security-audited in GitHub Actions (deliberately

tests/sparql_qc/README.md

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# QC SPARQL meta-test
2+
3+
This directory proves that the QC violation queries in `src/sparql/*-violation.sparql`
4+
can actually fail. It exists because they once could not.
5+
6+
It lives here, under `tests/`, on purpose: the ODK-managed tree
7+
(`src/ontology/`, `src/sparql/`) holds only ODK-owned scaffolding and the
8+
ontology itself. This harness only *reads* the queries under `src/sparql/`; it
9+
writes nothing there.
10+
11+
## The problem it guards against
12+
13+
The violation queries restrict to METPO terms with a namespace filter:
14+
15+
```sparql
16+
FILTER(isIRI(?term) && STRSTARTS(STR(?term), "https://w3id.org/metpo/"))
17+
```
18+
19+
That literal was once written `https://w3id.org/METPO_`, which matches no real
20+
METPO IRI. Every namespace-filtered check then returned zero rows and reported
21+
"0 violations" while CI stayed green for a long time (berkeleybop/metpo#487,
22+
#465). A check that cannot fail is worse than no check: it is false confidence.
23+
24+
The permanent root-cause fix is the single `namespaces:` declaration in
25+
`src/ontology/metpo-odk.yaml` (#465), which makes ODK regenerate the filters
26+
consistently instead of each query hand-writing the IRI prefix. This meta-test
27+
is the deterministic backstop that catches any future regression regardless of
28+
how it is introduced.
29+
30+
## The contract (read this before editing any check)
31+
32+
**Every `*-violation.sparql` must have at least one matching case in
33+
`qc-positive-fixture.ttl`.** The fixture is a tiny RDF graph seeded with one
34+
deliberate violation per query, on real `https://w3id.org/metpo/` IRIs.
35+
`run_sparql_metatest.sh` runs each query against it and fails if any returns
36+
zero rows.
37+
38+
So:
39+
40+
- **Adding a violation query?** Add a seeded case to the fixture in the same PR.
41+
The runner discovers queries by glob, so a new query with no fixture case
42+
fails the meta-test until you add one. This is intentional: a check cannot
43+
enter the repo without a known-failing input.
44+
- **Editing a violation query?** Re-run the meta-test and confirm its fixture
45+
case still trips it. If you tighten a filter, make sure the fixture still
46+
represents something the check is meant to catch.
47+
48+
The fixture IRIs are fixture-only (`9999xxx` range, never allocated) and live
49+
only in this directory; they are never merged into the ontology.
50+
51+
## Ontology-header check (#502)
52+
53+
`ontology-header-check.sparql` is a QC check kept here rather than as an edit to
54+
an ODK query. The runner runs it twice: against the bad-header node in
55+
`qc-positive-fixture.ttl` (must match >= 1 row, so it is proven able to fail)
56+
and against the real `metpo.owl` (must match 0 rows). It flags a creator or
57+
contributor given as a literal placeholder instead of an agent IRI, and a
58+
`dcterms:modified` that disagrees with `owl:versionInfo`. Deprecated DC Elements
59+
1.1 is already caught by ODK's `dc-properties-violation` and is not repeated.
60+
61+
## Running it
62+
63+
In the ODK container, from `src/ontology`:
64+
65+
```sh
66+
sh run.sh bash ../../tests/sparql_qc/run_sparql_metatest.sh
67+
```
68+
69+
CI runs it on every push/PR via `.github/workflows/sparql-metatest.yml`.
Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Ontology-header metadata QC (berkeleybop/metpo#502).
2+
#
3+
# This check lives OUTSIDE the ODK-managed tree on purpose (it does not edit any
4+
# ODK query). The meta-test runner runs it against the real metpo.owl, where it
5+
# must return 0 rows, and against qc-positive-fixture.ttl, where it must return
6+
# >= 1 row (so the check is proven able to fail).
7+
#
8+
# It returns a row for each problem on the ontology node:
9+
# (1) creator/contributor given as a literal instead of an agent IRI (ORCID);
10+
# (2) dcterms:modified disagreeing with the automated owl:versionInfo.
11+
# The header drifted unchecked until #495 (placeholder creator, fake
12+
# dcterms:issued, dcterms:modified contradicting owl:versionInfo). The deprecated
13+
# DC Elements 1.1 namespace is already caught by ODK's dc-properties-violation
14+
# (which scopes to https://w3id.org/metpo/ subjects, including the ontology node),
15+
# so it is not repeated here.
16+
17+
PREFIX owl: <http://www.w3.org/2002/07/owl#>
18+
PREFIX dcterms: <http://purl.org/dc/terms/>
19+
20+
SELECT DISTINCT ?ontology ?predicate ?value ?problem WHERE {
21+
# Scope to the METPO ontology node only, so a merged/imported ontology header
22+
# in the input cannot raise a false positive (per Copilot review on #506).
23+
?ontology a owl:Ontology .
24+
FILTER(isIRI(?ontology) && STRSTARTS(STR(?ontology), "https://w3id.org/metpo/"))
25+
{
26+
# (1) creator/contributor that is not an agent IRI (a literal placeholder, or
27+
# a blank node) -- anything non-IRI fails the "must be an ORCID" rule.
28+
VALUES ?predicate { dcterms:creator dcterms:contributor }
29+
?ontology ?predicate ?value .
30+
FILTER(!isIRI(?value))
31+
BIND("creator/contributor must be an agent IRI (e.g. ORCID), not a literal placeholder" AS ?problem)
32+
}
33+
UNION
34+
{
35+
# (2) dcterms:modified disagreeing with the automated owl:versionInfo
36+
?ontology dcterms:modified ?value ;
37+
owl:versionInfo ?vi .
38+
FILTER(STR(?value) != STR(?vi))
39+
BIND(dcterms:modified AS ?predicate)
40+
BIND(CONCAT("dcterms:modified '", STR(?value),
41+
"' disagrees with owl:versionInfo '", STR(?vi), "'") AS ?problem)
42+
}
43+
}
Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Positive-control fixture for the QC SPARQL violation queries.
2+
#
3+
# Guardrail for berkeleybop/metpo#500. Each subject below is a METPO-namespaced
4+
# IRI (https://w3id.org/metpo/<digits>) seeded with one deliberate violation case
5+
# for one query, so that query MUST return >= 1 row. (A query may emit more than
6+
# one row for a single case -- e.g. multiple-replaced_by joins the value pair in
7+
# both orderings -- which is fine; the meta-test only requires at least one.)
8+
#
9+
# Why this exists: four of the five violation queries filter terms with
10+
# STRSTARTS(str(?term), "https://w3id.org/metpo/")
11+
# That filter was once written as "https://w3id.org/METPO_", which matches no
12+
# real METPO term, so every check silently reported "0 violations" (#487, #465).
13+
# A check that cannot fail is false confidence. run_sparql_metatest.sh asserts
14+
# each query still matches a known-bad term in the real namespace.
15+
#
16+
# These IRIs are fixture-only (9999xxx range, never allocated); they live only
17+
# in this file and are never merged into the ontology.
18+
19+
@prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#> .
20+
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
21+
@prefix owl: <http://www.w3.org/2002/07/owl#> .
22+
@prefix dce: <http://purl.org/dc/elements/1.1/> .
23+
@prefix dcterms: <http://purl.org/dc/terms/> .
24+
@prefix IAO: <http://purl.obolibrary.org/obo/IAO_> .
25+
@prefix metpo: <https://w3id.org/metpo/> .
26+
27+
# -> dc-properties-violation.sparql : deprecated DC Elements 1.1 predicate on a METPO term
28+
metpo:9999001 a owl:Class ;
29+
rdfs:label "fixture: deprecated dc-elements predicate" ;
30+
dce:creator "should be dcterms:creator" .
31+
32+
# -> ontology-header-check.sparql : a bad ontology header.
33+
# (1) creator as a literal placeholder, not an agent IRI;
34+
# (2) dcterms:modified disagreeing with owl:versionInfo.
35+
# Uses dcterms: (not the dc-elements namespace) and dcterms:creator (not
36+
# contributor), so it trips only the header check, not the ODK dc-properties
37+
# or iri-range queries.
38+
<https://w3id.org/metpo/fixture-ontology> a owl:Ontology ;
39+
dcterms:creator "METPO Development Team" ;
40+
dcterms:modified "2024-01-01" ;
41+
owl:versionInfo "2026-06-05" .
42+
43+
# -> iri-range-violation.sparql : a property whose value must be an IRI, given a literal
44+
metpo:9999002 a owl:Class ;
45+
rdfs:label "fixture: contributor as a literal, not an IRI" ;
46+
dcterms:contributor "Jane Doe" .
47+
48+
# -> label-with-iri-violation.sparql : an rdfs:label containing a URL
49+
metpo:9999003 a owl:Class ;
50+
rdfs:label "fixture: a label that wrongly embeds http://example.org/foo" .
51+
52+
# -> multiple-replaced_by-violation.sparql : two distinct IAO:0100001 (term replaced by) values
53+
metpo:9999004 a owl:Class ;
54+
rdfs:label "fixture: two different replacements" ;
55+
IAO:0100001 metpo:1000000 , metpo:2000000 .
56+
57+
# -> owldef-self-reference-violation.sparql : equivalent-class definition that references itself
58+
metpo:9999005 a owl:Class ;
59+
rdfs:label "fixture: self-referential equivalent class" ;
60+
owl:equivalentClass [
61+
a owl:Class ;
62+
owl:intersectionOf ( metpo:9999005 )
63+
] .
Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,96 @@
1+
#!/usr/bin/env bash
2+
# Meta-test for the QC SPARQL violation queries (berkeleybop/metpo#500).
3+
#
4+
# Runs every src/sparql/*-violation.sparql against the positive-control fixture
5+
# (qc-positive-fixture.ttl) and asserts each returns >= 1 row. This proves the
6+
# checks actually MATCH METPO terms: four of them filter on the term namespace
7+
# with STRSTARTS(str(?term), "https://w3id.org/metpo/"), and that filter was
8+
# once "https://w3id.org/METPO_", matching nothing, so every check silently
9+
# reported "0 violations" (#487, #465). A check that cannot fail is worse than
10+
# no check.
11+
#
12+
# This harness lives OUTSIDE the ODK-managed tree (src/ontology, src/sparql): it
13+
# only READS the queries under src/sparql. Queries are discovered dynamically,
14+
# so a new *-violation.sparql with no seeded fixture case fails here, which
15+
# forces the fixture to stay complete.
16+
#
17+
# Run locally from src/ontology:
18+
# sh run.sh bash ../../tests/sparql_qc/run_sparql_metatest.sh
19+
# Requires: robot on PATH (present in the obolibrary/odkfull container).
20+
set -euo pipefail
21+
22+
HERE="$(cd "$(dirname "$0")" && pwd)"
23+
REPO_ROOT="$(cd "$HERE/../.." && pwd)"
24+
SPARQLDIR="$REPO_ROOT/src/sparql"
25+
FIXTURE="$HERE/qc-positive-fixture.ttl"
26+
OUT="$(mktemp -d)"
27+
trap 'rm -rf "$OUT"' EXIT
28+
29+
shopt -s nullglob
30+
queries=("$SPARQLDIR"/*-violation.sparql)
31+
if [ "${#queries[@]}" -eq 0 ]; then
32+
echo "::error::no *-violation.sparql queries found in $SPARQLDIR"
33+
exit 1
34+
fi
35+
36+
fail=0
37+
for q in "${queries[@]}"; do
38+
name="$(basename "$q")"
39+
res="$OUT/$name.tsv"
40+
robot query --input "$FIXTURE" --query "$q" "$res"
41+
rows=$(($(wc -l < "$res") - 1)) # the first line is the SELECT header
42+
if [ "$rows" -ge 1 ]; then
43+
echo "OK $name matched $rows fixture violation(s)"
44+
else
45+
echo "::error::$name matched 0 rows against the positive-control fixture: the check is a silent no-op (e.g. wrong term-namespace filter)."
46+
fail=1
47+
fi
48+
done
49+
50+
if [ "$fail" -ne 0 ]; then
51+
echo
52+
echo "One or more QC violation queries matched nothing they are meant to catch."
53+
echo "Fix: seed the missing case in tests/sparql_qc/qc-positive-fixture.ttl, or"
54+
echo "correct the query's term-namespace filter (https://w3id.org/metpo/)."
55+
exit 1
56+
fi
57+
58+
echo "All ${#queries[@]} QC violation queries matched their positive-control fixture."
59+
60+
# ---------------------------------------------------------------------------
61+
# Ontology-header QC (berkeleybop/metpo#502). This check is maintained here,
62+
# outside the ODK tree, rather than as an edit to an ODK query. It must:
63+
# (a) match >= 1 row against the bad-header fixture (proven able to fail), and
64+
# (b) match 0 rows against the real released metpo.owl (the actual assertion).
65+
# ---------------------------------------------------------------------------
66+
echo
67+
HEADER_Q="$HERE/ontology-header-check.sparql"
68+
REAL_OWL="$REPO_ROOT/metpo.owl"
69+
70+
robot query --input "$FIXTURE" --query "$HEADER_Q" "$OUT/header-fixture.tsv"
71+
hrows=$(($(wc -l < "$OUT/header-fixture.tsv") - 1))
72+
if [ "$hrows" -ge 1 ]; then
73+
echo "OK ontology-header-check.sparql matched $hrows fixture violation(s)"
74+
else
75+
echo "::error::ontology-header-check.sparql matched 0 rows against the fixture: the check is a silent no-op."
76+
fail=1
77+
fi
78+
79+
if [ -f "$REAL_OWL" ]; then
80+
robot query --input "$REAL_OWL" --query "$HEADER_Q" "$OUT/header-real.tsv"
81+
rrows=$(($(wc -l < "$OUT/header-real.tsv") - 1))
82+
if [ "$rrows" -le 0 ]; then
83+
echo "OK ontology-header-check.sparql found no problems in metpo.owl"
84+
else
85+
echo "::error::the ontology header in metpo.owl has $rrows metadata problem(s):"
86+
cat "$OUT/header-real.tsv"
87+
fail=1
88+
fi
89+
else
90+
echo "::error::expected released ontology at $REAL_OWL but it is missing."
91+
fail=1
92+
fi
93+
94+
if [ "$fail" -ne 0 ]; then
95+
exit 1
96+
fi

0 commit comments

Comments
 (0)