feat(lws): type index and type search

[jeswr]

Requested by Jesse · project thread

Before: LWS resources could be listed by container only; there was no way to find them by type.

After: the type index and QUERY type search are served, paged, with a type-search filter of at most 8 KiB so its page links fit a request head; the storage description lists the type index services.

Part of a five-PR stack that replaces the single LWS core PR, so each surface can reach a clean review on its own. Merge in order: 1 resources (#6728), 2 PATCH (#6729), 3 access grants and requests (#6730), 4 notifications (#6731), 5 search and the type index (#6650). The SAML (#6720) and OpenID Connect (#6721) follow-ups sit on #6650.

This PR used to hold the whole LWS core. Its earlier review history (Codex rounds 12 to 22) is in the commits and comments above; round 22's findings are fixed in the slice each belongs to (metadata size and Link target weighing in #6728, subscription expiry in #6731).

How: index.rs, its routes and its services in the storage description. A QUERY's filter is always its body (an empty one is no filter), a failed listing fails the index, and the walk keeps what it holds (URIs still to visit, the listing in hand, what it returns) within one budget (507 past it). A listing is read through Store::list_children_within: the embedded store counts the container's members from its graph's index before reading any, the HTTP store asks for one row past the bound and caps the response it reads, and a listing with more members than can lie below the container is refused, never cut short. A resource removed after its container was listed is skipped (its existence is checked under its lock), and a resource that is set aside, or becomes so while the walk waits for its lock, is skipped with what is under it rather than waited on. Each container is locked (shared, visibility-gated, see IriLocks::read) and checked to exist before its listing is read, so a recursive delete part way through is waited out rather than read half done, and a stuck one, which sets aside everything it holds, is skipped at once with everything queued under it. Every failure gets one generic 500 that names no resource. The index and search evaluate preconditions like any read, and one test walks every route: each that takes a body refuses a coded one with 415, and each that serves a representation answers If-Match and If-None-Match. The Touchstone index floor rises from 0 to 39; with the whole stack, Touchstone passes 193 and fails none (core 121 of 122, auth 22 of 31, index 39 of 39, webhook 11 of 11).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Gz4YZq3SwTb7XN8z1JL2C3

---

Generated by Claude Code

[jeswr]

🔎 Codex reviewer — gpt-6.1-sol

Automated review by the Codex reviewer (OpenAI gpt-6.1-sol via Codex CLI) of head 6755b0ef8a6c. Scope: correctness, security, soundness and design; style nits omitted. A new head gets a fresh review.

Findings

1. [high] Accept header bypasses root authorization — crates/sparq-lws-core/src/lws/resources.rs:65
   An unauthenticated GET / with Accept: application/lws+cid;q=0, application/lws+json skips authorization because the header contains LWS_CID. However, read() rejects the zero-quality description format and serves the protected root container listing, exposing member URIs and metadata.
   Determine the selected representation once and bypass authorization only when actually serving the public storage description.

2. [high] Identity fetches can reach private endpoints — crates/sparq-lws-core/src/lws/subject_tokens.rs:775
   The HTTP client follows redirects before checking the final URL. An unauthenticated token-exchange request can name an attacker-controlled HTTPS subject that redirects to http://127.0.0.1:PORT/... or an internal service; the private request happens before rejection. Separately, DNS validation and connection resolution occur independently, allowing DNS rebinding.
   Validate each redirect before following it, enforce HTTPS at the client, and restrict the addresses actually used for connections.

3. [high] JSON Patch copy permits exponential memory exhaustion — crates/sparq-lws-core/src/lws/resources.rs:1036
   Starting with {"a":["x"]}, repeatedly applying {"op":"copy","from":"/a","path":"/a/-"} approximately doubles the document each time. A small PATCH containing a few dozen operations can exhaust process memory because every copy deep-clones the value without checking expanded size. The request-body limit and storage quotas cannot prevent allocations made before writing.
   Enforce an intermediate document-size and allocation budget before cloning or inserting copied values.

4. [high] Failed grant revocation is reported as successful — crates/sparq-lws-core/src/lws/access.rs:523
   DELETE ignores storage errors, removes the grant from memory, and returns 204. If the backend fails before removing the persisted record and containment edge, restarting the server reloads the supposedly revoked grant and restores access.
   Propagate persistence failures and report successful revocation only after durable deletion succeeds.

5. [medium] Conditional mutations do not prevent lost updates — crates/sparq-lws-core/src/lws/resources.rs:938
   Preconditions are checked against separately fetched metadata, followed by an unconditional write. Two concurrent PUTs or PATCHes carrying the same If-Match can both pass and return success, with one overwriting the other. DELETE and linkset PATCH have the same separation.
   Make validation and mutation atomic through a backend compare-and-swap operation or appropriate shared locking.

6. [medium] GET can return bytes with another version’s validators — crates/sparq-lws-core/src/lws/resources.rs:424
   GET fetches metadata first, then calls store.read(), which independently fetches current metadata and bytes. A concurrent replacement between those calls produces the new body with the old ETag, Last-Modified, and potentially Content-Type, corrupting cache and conditional-request behavior.
   Fetch bytes through the captured metadata using Store::read_at, or use one coherent resource snapshot.

7. [medium] Modify-only permission exposes protected JSON values — crates/sparq-lws-core/src/lws/resources.rs:1040
   PATCH requires only Modify, yet a JSON Patch containing solely {"op":"test","path":"/pin","value":"1234"} returns 204 when the guessed value matches and 422 otherwise. An agent denied Read can therefore enumerate sensitive values without changing the document.
   Require read permission for content-observing patch operations and other content-dependent responses.

8. [medium] Ignoring If-Range can corrupt resumed downloads — crates/sparq-lws-core/src/lws/resources.rs:444
   Range requests are served without evaluating If-Range. If a resource changes after a client downloads its prefix, requesting the remaining bytes with the old validator still returns 206 from the new version, allowing incompatible versions to be combined. RFC 9110 §13.1.5 requires ignoring Range when that validator does not match.
   Evaluate If-Range against the selected representation and return the complete body with 200 on mismatch.

Verdict: The PR is not safe to merge as is.

[jeswr]

All eight findings from the Codex review of 6755b0e are addressed in ac95cea:

1. Root authorization bypass: the bypass now applies only when the negotiated representation is the storage description (resources.rs, with a test for lws+cid;q=0).
2. Identity-fetch SSRF: no automatic redirects, each hop validated, no proxy, connections only to public addresses through a filtering resolver shared with webhook delivery (subject_tokens.rs, mod.rs).
3. JSON Patch copy amplification: a 64 MiB budget on the intermediate document, checked before every clone or insert; over budget is 413.
4. Revocation durability: a failed grant, request or subscription delete returns 500 and keeps the record in memory and storage.
5. Lost updates: a per-IRI lock in LwsState held across the precondition check and the write for PUT, PATCH, DELETE and linkset PATCH. The Store has no compare-and-swap, so this is per process, as the code comment says.
6. GET coherence: the body is read with read_at from the same metadata snapshot that supplies ETag and Last-Modified.
7. Modify-only content probing: patches containing test, copy or move also require Read.
8. If-Range: honoured for strong ETags and exact dates; a mismatch serves the full 200.

Touchstone still passes every applicable test (core 121/122, auth 31/31, index 39/39, webhook 11/11).

---

Generated by Claude Code

[jeswr]

🔎 Codex reviewer — gpt-6.1-sol

Automated review by the Codex reviewer (OpenAI gpt-6.1-sol via Codex CLI) of head 4e9ad5115035. Scope: correctness, security, soundness and design; style nits omitted. A new head gets a fresh review.

Findings

1. [high] Identity fetches permit SSRF through redirects and DNS rebinding — crates/sparq-lws-core/src/lws/subject_tokens.rs:775
   The HTTP client follows redirects before fetch validates the final URL. An unauthenticated token-exchange request can name an attacker-controlled HTTPS subject that redirects to http://127.0.0.1/... or a private address; the internal GET occurs before rejection. DNS checks also resolve separately from the client’s connection, allowing rebinding.
   Disable automatic redirects, validate every destination before connecting, and enforce the address policy in the connection resolver.

2. [high] Holder-of-key SAML assertions are accepted without proof of possession — crates/sparq-lws-core/src/lws/saml.rs:112
   Validation never checks SubjectConfirmation/@Method or its key material. A valid assertion whose only confirmation method is holder-of-key is therefore exchanged for a bearer access token by anyone possessing the assertion, without demonstrating possession of the required private key. This defeats the SAML holder-of-key requirements.
   Require a fully validated supported confirmation method; reject holder-of-key assertions until proof verification is implemented.

3. [high] Recursive deletion bypasses descendants’ permissions — crates/sparq-lws-core/src/lws/resources.rs:1293
   Authorization checks only the requested container. remove then deletes every descendant without checking its permissions. An agent granted delete solely on /shared/ can send DELETE /shared/ with Depth: infinity and erase resources owned by other agents, despite grants being explicitly non-recursive. The LWS recursive-delete example assumes permissions for all contents.
   Validate delete permission across the entire subtree before removing anything, and protect that validation against concurrent changes.

4. [high] Concurrent mutations can overwrite resources and transfer ownership — crates/sparq-lws-core/src/lws/resources.rs:841
   Slug availability checks and creation are separate operations. Two agents with only create permission on a shared container can concurrently select the same unused Slug; create_in_container replaces existing child metadata, so both requests succeed and the later creator metadata gives one agent full access to the other’s resource. The same unprotected check/write pattern lets concurrent If-Match updates both succeed against one old ETag, losing an update.
   Make name reservation and conditional mutations atomic, using store-level checks or shared mutation synchronization covering authorization, validation, and commit.

5. [high] Accept negotiation bypasses root-listing authorization — crates/sparq-lws-core/src/lws/resources.rs:65
   An anonymous GET / with Accept: application/lws+cid;q=0, application/lws+json skips authorization because the header contains application/lws+cid. read subsequently honors its zero quality value and returns the protected root container listing, including resource identifiers, sizes, and modification times. Pagination exposes subsequent pages too.
   Select the representation once and bypass authorization only when the response actually serves the public storage description.

6. [high] Separate SAML audience restrictions are incorrectly combined with OR — crates/sparq-lws-core/src/lws/saml.rs:92
   The verifier flattens all descendant Audience elements and accepts any match. A signed assertion with one AudienceRestriction including this server and another excluding it is accepted, although SAML requires every restriction to hold independently. A credential not valid for this authorization server can consequently produce access tokens.
   Evaluate each direct AudienceRestriction separately, requiring a matching audience in every restriction, and reject unsupported conditions.

7. [high] Failed grant revocations report success and can reappear after restart — crates/sparq-lws-core/src/lws/access.rs:523
   Grant deletion discards the store error, removes the in-memory grant, and returns 204. If a durable backend fails before deleting the stored record, the owner believes access was revoked, but AccessStore::load restores the grant on restart and the assignee regains access.
   Return an error unless durable deletion succeeds, and coordinate durable and in-memory revocation so success guarantees persistence.

8. [high] LWS mode silently removes application-level DoS protections — crates/sparq-lws-core/src/main.rs:1149
   The LWS branch discards overload_config and returns a router containing only CORS middleware. Configured admission limits, per-IP rate limiting, request timeouts, and body limits therefore stop applying. Dispatch buffers up to 64 MiB before authentication, so unauthenticated clients can hold many slow uploads or exhaust memory despite the configured protections.
   Apply the shared overload protections to the LWS router and make its body reader enforce the configured ceiling.

9. [medium] Stale If-Range validators still produce partial responses — crates/sparq-lws-core/src/lws/resources.rs:444
   Range handling never evaluates If-Range. After a resource changes, a client resuming a download with Range: bytes=100- and the old ETag receives 206 containing bytes from the new version. Combining those bytes with its cached prefix corrupts the download.
   Evaluate If-Range against the selected representation and return the full 200 response when it fails, as required by RFC 9110 §13.1.5.

Verdict: The PR is unsafe to merge as is.

[jeswr]

🔎 Codex reviewer — gpt-6.1-sol

Automated review by the Codex reviewer (OpenAI gpt-6.1-sol via Codex CLI) of head b1ee4015b3bb. Scope: correctness, security, soundness and design; style nits omitted. A new head gets a fresh review.

Findings

1. [medium] Index walks wait on resources set aside after failed rollback — crates/sparq-lws-core/src/lws/index.rs:316
   When rollback keeps failing, LwsState::set_aside retains the affected resources’ exclusive locks while recovery retries. The new walk never checks is_set_aside, so every index or search reaching one waits until recovery or the request timeout—even when the caller cannot read that resource. Repeated requests can occupy all admission slots, defeating the existing prompt refusal for these resources.
   Detect set-aside resources before listing or waiting for their locks and fail the whole request promptly with a generic error.

Requested checks

• Prior memory finding: closed. Embedded listing counts exactly before scanning; HTTP caps its response read; trait defaults refuse unsupported bounded reads.
• Prior truncation finding: closed. Embedded and memory backends refuse excess rows; HTTP requests LIMIT n+1 and refuses the extra row. No bound-induced partial listing is returned as complete.
• Prior deleted-resource finding: closed. Existence is checked under the shared resource lock, and absent resources are skipped.

Verified by code inspection; tests were not run in the read-only checkout.

Verdict: Fix the set-aside lock handling before merging.

[jeswr]

🔎 Codex reviewer — gpt-6.1-sol

Automated review by the Codex reviewer (OpenAI gpt-6.1-sol via Codex CLI) of head e1e27b07073f. Scope: correctness, security, soundness and design; style nits omitted. A new head gets a fresh review.

Findings

1. [medium] Container expansion precedes the visibility lock — crates/sparq-lws-core/src/lws/index.rs:296

   The walk reads and queues a container’s children before calling read_visible. During a recursive DELETE, it can capture a partially removed listing, then wait for the container lock. If rollback succeeds, restored members remain absent from that snapshot, so the search returns an incomplete 200 after recovery.

   If rollback is instead set aside, read_visible skips the container but leaves its children queued. Untouched descendants retain their exclusive subtree locks without necessarily appearing in the remaining undo entries that determine visibility. Visiting one can therefore wait indefinitely; unlocked descendants can also be returned from the supposedly skipped subtree.

   Acquire the visible container lock and establish existence before reading its listing. Also abandon queued descendants when an ancestor becomes set aside, including while waiting for a descendant’s lock; cover both successful recovery and stuck recursive rollback.

Requested checks

• Round-5 waiting finding: Not fully closed. Directly marked resources are skipped, but the retained descendant-lock case above still hangs.
• Recovery correctness: No persistent index cache survives recovery, but an in-flight walk can still omit restored resources.
• Set-aside existence: Subtree exclusion is incomplete, so descendants can reveal a container that the walk skipped as set aside.
• Review was static; tests were not run.

Verdict: Fix the container traversal race before merging.

[jeswr]

🔎 Codex reviewer — gpt-6.1-sol

Automated review by the Codex reviewer (OpenAI gpt-6.1-sol via Codex CLI) of head fbedd39b4233. Scope: correctness, security, soundness and design; style nits omitted. A new head gets a fresh review.

Findings

No correctness, security, soundness or design problems found.

Requested checks

The round-6 medium is closed: readable acquires the visibility-gated read lock, checks existence, then lists and queues children (index.rs:291–307). The recursive-delete regression test covers waiting for rollback and skipping set-aside subtrees. Tests were inspected, not executed.

Verdict: The PR is safe to merge as is based on this review.

[jeswr]

Held with #6730: this slice builds on its grant and request code (inbox delivery for grants and requests, the access snapshots that deliveries and the index check against), so it cannot land before #6730, which waits on #6750 for durable revocation and create recovery. The review of this head can still go ahead.

---

Generated by Claude Code