build(release): crates.io-ready publish set (12 crates + sparq-spargebra fork)

[jeswr]

Requested by Jesse · project thread

Summary

Before: nothing from sparq was publishable to crates.io in practice. The registry closure was 37 crates (28 reached from the core alone through optional research features like ZK, trust, policy and terse), and published crates would have built against unpatched upstream spargebra, silently losing the SPARQ-PATCHES.md fixes, including the §8 recursion-depth cap that guards the unauthenticated /sparql endpoint.

After: a deliberate 12-crate publish set plus the parser fork. cargo publish --workspace --dry-run packages and verifies all 13 in dependency order, and the bootstrap list in docs/release.md §4 is the exact command sequence.

How

• Publish set (12): sparq-core, -substrate, -jsonld, -engine-serialize, -engine-service, -engine, -reason, -shacl, -hdt, -serve, -server, -cli. The 25 other formerly-publishable crates are now publish = false. release-plz.toml's version group is those 12.
• spargebra fork → sparq-spargebra: vendor/spargebra is renamed to the package sparq-spargebra (library name still spargebra, so no source changes). The workspace dependency is spargebra = { package = "sparq-spargebra", version = "0.4.6", path = "vendor/spargebra" }. Third-party crates that name upstream spargebra (oxigraph/spareval/sparopt in the bench and differential harnesses) are routed to the same fork through an unpublished re-export shim, vendor/spargebra-shim, because Cargo matches [patch] by package name. The 13 detached bench/zk workspaces point their own [patch.crates-io] spargebra at the same shim, so direct spargebra users there (e.g. bench/serve's parse spikes) also get the fork. It is versioned independently and published by hand when it changes.
• scripts/publish-strip.py: at packaging time it removes optional dependencies on publish = false crates from the publishable manifests, together with every feature that needs them (transitively), orphaned optional deps, dev-deps on unpublished crates, and gated examples. It trims the workspace to the publish set, and with --with-vendored it adds the fork. It refuses a non-optional edge to an unpublished crate, a stripped default feature, or a published crate enabling another's stripped feature. --check prints the plan. Today it strips el/terse from sparq-cli and complete, facets, federation-descriptors, geo, http3, odrl-authz, solid-authz, solid-authz-trust, terse from sparq-server. Those stay available from a git build. No default feature is affected.
• Guard: release-interval-guard.py now allows optional and dev edges to unpublished crates, since the strip removes them. Non-optional edges still refuse.
• publish.yml crates job: runs the strip, then one cargo package --workspace --no-verify --allow-dirty. The old per-crate cargo package -p loop could not resolve unpublished workspace deps. tomlkit is hash-pinned in .github/requirements/publish-strip.txt.
• Runbook: docs/release.md §4 has the publish set and the bootstrap sequence. It notes the crates.io new-crate rate limit (a short burst, then about one new name per 10 minutes, so about 90 minutes for 13). §8c has the release-plz flip, which adds the strip step before release-plz release.
• Book: a short "Rust crates" page listing the set, with rows tested against the manifests. This ports the useful part of the closed docs(docs): Backfill: existing published crates have no website page and no guide chapter #6148.

Why strip instead of a restructure: the research edges are #[cfg(feature)] blocks inside sparq-server (http.rs, solid_authz.rs) and sparq-cli. Inverting them into separate crates means moving a lot of code that the LWS and ZK threads are actively changing. The strip keeps every in-repo feature working, and its refusal rules keep it sound. To publish a capability later, flip its crate out of publish = false and add it to the version group. The strip then stops removing it.

Overlap with #6649: merges cleanly on top of it, and with #6649 the strip no longer needs to remove sparq-server's zk/trust dev-deps.

Follow-up (not here): upstream spargebra 0.4.7 is now out. §7–§10 are sparq-local, so the fork stays; rebasing it onto 0.4.7 is separate work.

Verified locally

• python3 scripts/publish-strip.py --with-vendored && cargo publish --workspace --dry-run --allow-dirty: all 13 crates packaged and verified, exit 0.
• The new publish.yml packaging step, run locally: 13 .crate files, matching the expected count.
• cargo check --workspace --exclude sparq-py: clean. cargo test --manifest-path vendor/spargebra/Cargo.toml: 28 passed (recursion-depth + custom-aggregate regressions).
• python3 -m unittest scripts/tests/test_release_publish_guard.py scripts/tests/test_publish_strip.py: 86 + 6 OK.
• release-interval-guard.py --dry-run: 12 crates, version group covers all 12.
• cargo metadata on bench/parse, bench/serve, zk/xpath/differential: resolve.

Base gate (always required)

• cargo build --workspace succeeds (cargo check --workspace --exclude sparq-py clean locally; CI runs the build).
• cargo clippy --workspace --exclude sparq-py --all-targets -- -D warnings: no Rust source changed; left to CI.
• The code this PR touches is formatted (manifests, Python, markdown).
• cargo test passes for every crate this PR touches (the vendored fork's tests).

Targeted re-evaluation (check the rows that apply to your change)

• Parser (spargebra): package rename only, with byte-identical sources. Lock diff adds just sparq-spargebra and keeps one parser in the graph via the shim. The conformance ratchet runs in CI.
• Cargo dependencies: Cargo.lock only renames the fork (no version changes). The spargebra cargo-vet policy/exemption still applies to the shim. cargo deny/vet run in CI.

Ratchets and conventions

• No ratchet lowered.
• No hard-coded performance numbers.
• Docs made false by this change are updated in the same change (release runbook, SPARQ-PATCHES.md, release-plz.toml, guard docstring).

Security

• No regression. This closes a gap: published crates now keep the parser recursion-depth DoS cap (SPARQ-PATCHES.md §8).

🤖 Generated with Claude Code

https://claude.ai/code/session_014tB5DfVUDWY537tfMuLEPi

---

Generated by Claude Code

[jeswr]

🔎 Codex reviewer — gpt-6.1-sol

Automated review by the Codex reviewer (OpenAI gpt-6.1-sol via Codex CLI) of head 1907d1c0379d. Scope: correctness, security, soundness and design; style nits omitted. A new head gets a fresh review.

Findings

1. [low] Serving spikes measure the wrong parser — bench/serve/Cargo.toml:18

   Adding package = "sparq-spargebra" makes this patch apply to the fork, while the direct dependency on line 26 still requests upstream spargebra. The updated lockfile confirms both resolve separately. Consequently, cache_spike and point_spike measure upstream parsing when estimating the parsing cost of engine queries, which use the patched fork.

   Rename the direct dependency to sparq-spargebra, or point the upstream patch at vendor/spargebra-shim, then regenerate the lockfile.

Verdict: Fix the parser-measurement regression before merging.

[jeswr]

Fixed in c03c286: the finding was right, and it applied to all 13 detached bench/zk workspaces, not only bench/serve. Each [patch.crates-io] spargebra now points at vendor/spargebra-shim, so direct spargebra users resolve to the fork. bench/serve's lock now has spargebra (shim) → sparq-spargebra and no registry spargebra. cargo metadata resolves for all 13.

---

Generated by Claude Code

[jeswr]

Local gate (GitHub Actions runners down; maintainer-approved local-gate procedure). Tested PR head c03c2860 merged with main cbabccec64117480b503ea7c0645064715a3462f, running ci-fast.yml's steps:

• clippy -D warnings on the core crates (--all-targets): pass
• cargo nextest run on the core crates with --profile ci: 3007 passed, 27 skipped
• doctests on the core crates: 38 passed, 0 failed
• W3C SPARQL conformance: 1225 pass / 0 fail / 4 documented divergences, for a total of 1229, which meets the ratchet (>= 1229)
• publish-strip.py --with-vendored followed by cargo publish --workspace --dry-run --allow-dirty: 13/13 crates packaged and verified
• scripts/tests/test_release_publish_guard.py and test_publish_strip.py: OK

The Codex finding (the bench parser patch) was fixed in c03c286.

---

Generated by Claude Code

[jeswr]

Main moved to f6457876b5e60505919851f501128b251fbe2367 (two npm lockfile bumps), so I re-ran the gate on PR head c03c2860 merged with it. Results were identical: clippy pass; nextest 3007 passed; doctests 38 passed; conformance 1225 + 4 = 1229 (meets the ratchet); publish dry-run 13/13; script tests OK.

---

Generated by Claude Code