1610_verify_integrity_of_high_value_payloads

.github/workflows/nightly-fuzz.yml

@@ -117,6 +117,12 @@ jobs:
           - name: driver-config-key-field-round-trip
             pkg: ./token/driver
             func: FuzzConfigKeyFieldRoundTrip
+          - name: storage-integrity-stored-token-request
+            pkg: ./token/services/storage/integrity
+            func: FuzzCheckStoredTokenRequest
+          - name: storage-integrity-token-request-actions
+            pkg: ./token/services/storage/integrity
+            func: FuzzCheckTokenRequestActions
 
     steps:
       - name: Checkout code

docs/README.md

@@ -17,6 +17,7 @@ Welcome to Panurus documentation.
 
 *   [**HTLC Deadlines and Clock Synchronisation**](security/htlc_deadline_clock_assumptions.md): The clock-synchronisation assumption that the HTLC claim/reclaim deadline rests on, and the deadline margin it requires of a deployment.
 *   [**Selector Resource Limits**](security/selector_resource_limits.md): How to throttle token selection by supplying a custom `Locker`.
+*   [**Store Integrity Verification**](security/store_integrity_verification.md): What each store verifies about the payloads it persists, what it requires of its caller, and what it deliberately does not check.
 
 ## Command-Line Tools
 

docs/security/store_integrity_verification.md

@@ -0,0 +1,154 @@
+# Store Integrity Verification
+
+The store services persist the payloads that Panurus later treats as evidence: the token request a
+transaction is re-validated against, the public parameters it is validated *under*, the identity a
+signer is bound to, the acknowledgement that a party endorsed a transaction. This page states, per
+asset class, **what a store verifies, what it requires of its caller, and what it deliberately does
+not check** — so that a reader of a store method knows which of the three applies without reading
+its implementation.
+
+The checks themselves live in one backend-agnostic package,
+[`token/services/storage/integrity`](../../token/services/storage/integrity), and each store method
+that applies one carries a `Verification:` clause in its Godoc naming it. Both are greppable on
+purpose: `grep -rn "Verification:"` enumerates the contract, and `grep -rn integrity.Check`
+enumerates the enforcement.
+
+## Scope
+
+This is a **structural** integrity boundary, not a second validator. The checks need only the bytes
+themselves plus the key those bytes are stored under; none of them performs I/O, consults a ledger,
+or evaluates a zero-knowledge proof. A token request is fully verified only by a `token.Validator`
+against a ledger, and that happens where it already happened — see
+[Endorsement responder security](../services/ttx_responder_security.md).
+
+What the boundary buys is that a payload which **could not have been produced by a correct caller**
+is refused rather than stored, and a payload which **could not honestly be attributed to the row it
+was found in** is reported rather than returned. The failures it catches are systemic ones: a
+truncated or garbled blob, a row swap, a wrong-row read on a hash-addressed table, a write that
+skipped the validating path, an empty value standing in for a real one.
+
+Out of scope: trust-chain and revocation validation, semantic validation of actions, and anything
+requiring a network round-trip from inside a store call.
+
+## Asset classes
+
+| Asset | At insert | On retrieval | Caller must have done |
+|---|---|---|---|
+| Token request (`ttxdb`, `auditdb` — `TokenRequestWithMetadata`) | Non-empty `tx_id`, non-empty request, non-empty `pp_hash` | Deserializes, declares a supported protocol version, and its **anchor equals the `tx_id` it is filed under** | Serialized it from a live `token.Request` (the marshal round-trip is what makes the insert-side re-parse redundant) |
+| Token request (`endorserdb` — bare actions and signatures) | Non-empty `tx_id`/request/`pp_hash`, deserializes at a supported version, **carries at least one action** | — (this format has no anchor; see below) | Validated it — `validator.UnmarshallAndVerifyWithMetadata`, and taken `pp_hash` from the *local* TMS, not from the peer |
+| Public parameters (`tokendb`) | Hash computed by the store from the bytes it is storing | **Recomputed and compared against the hash the row is filed under** | — |
+| Identity (`identitydb`, both backends) | Non-empty | Non-empty, and the stored identity **compared against the requested one** (`GetAuditInfo`, `GetTokenInfo`, and `GetSignerInfo` on SQL) | Matched it against its audit info, and — at `wallet.Service` — established that an owner verifier can be derived from it |
+| Signer registration (`token.SignatureService`) | Identity non-empty **and** a verifier derivable from it in some role | — | — |
+| Endorsement acknowledgement (`tx_ends`) | Non-empty endorser, non-empty signature | — (the signed message is not persisted; see below) | **Verified `sigma` against the exact payload it sent to that party** — `CollectEndorsementsView.distributeTxToParty` does |
+| Movements, transaction records, token locks, application/public metadata, audit info blobs, `IdentityConfiguration` raw | — | — | Everything: these are stored as given |
+
+The last row is a deliberate posture, not an omission. These classes are high-volume, individually
+low-value, and — decisively — have **no cheap self-consistency predicate**: there is no field in a
+movement row that a structural check could disagree with. Adding a check with no predicate to
+evaluate would cost write throughput and catch nothing.
+
+## Requirements
+
+1. **Callers of `endorserdb.AppendValidationRecord` MUST validate the token request first**, and
+   MUST take `pp_hash` from their own TMS rather than from the peer that sent the request. The store
+   checks that the payload is a deserializable request with at least one action; it cannot check
+   that the actions are legal.
+2. **Callers of `AddTransactionEndorsementAck` MUST verify the signature** against the payload they
+   sent to that endorser, before storing. The store checks only that an endorser and a signature are
+   present.
+3. **No caller may pass an empty identity** to the identity store. See the next section for why this
+   is a correctness requirement and not a tidiness one.
+4. **The checks MUST remain unconditional.** No functional option, no setter, no configuration key,
+   no build tag may turn one off. This is enforced by
+   [`nobypass_test.go`](../../token/services/storage/integrity/nobypass_test.go), which reads the
+   source of every package applying a check and fails on an identifier named like a bypass, on a
+   variadic parameter added to a check, on mutable package-level state in the `integrity` package,
+   on a verification-related configuration key, and on a check whose error is discarded.
+
+## What is deliberately not checked, and why
+
+Three of these are structural limits rather than choices, and each one is a place where the obvious
+check does not exist to be added.
+
+**`pp_hash` is not a digest of the request.** It is the hash of the *public parameters* the request
+was created under. It therefore provides no integrity for the `request` column, and no
+retrieval-time "does the stored hash match the stored bytes" check is possible for token requests.
+What *is* available, and is what the retrieval-side check uses, is the request's own **anchor**: the
+transaction id the request commits to, covered by the signatures inside it. Comparing the anchor
+against the `tx_id` the row is keyed by is what ties the bytes to the row they were found in, and it
+catches truncation, encoding drift, and row swaps. The bare actions-and-signatures format held by
+`endorserdb` carries no anchor at all, so records in that format cannot be bound to their
+transaction id by a structural check — only by the caller's validation.
+
+**The message an endorsement acknowledgement signs is not persisted.** `tx_ends` holds
+`(endorser, sigma)`. The signed message is the *per-party filtered* transaction payload, different
+for each endorser, and it is not stored anywhere. Retrieval-time re-verification of an ack therefore
+has nothing to verify against, and adding it requires a schema change — a persisted digest of the
+endorsed message — which is tracked separately. Until then, the posture is: verified at insert by
+the only producer, contract-documented as not re-verifiable on read. The store does refuse an empty
+endorser or an empty signature, which matters more than it looks:
+`ttx.TransactionInfo` presents acks as a map keyed by endorser and its consumers do not inspect the
+values, so a row with an empty signature reads as "this party signed".
+
+**A supplied verifier is not compared against the identity it is registered for.** `driver.Verifier`
+exposes only `Verify(message, sigma)` — there is no canonical public key to compare, so establishing
+agreement would require a new accessor implemented across every identity type (x509, idemix,
+htlc, multisig, boolpolicy). What `token.SignatureService.RegisterSigner` enforces instead is the
+substantive, non-tautological part: the identity is non-empty, and *some* verifier can be derived
+from it by this driver. That rules out binding a signer to bytes no verifier can be built from —
+an identity that can sign but whose signatures nothing can check. The comparison itself would be a
+tautology for the in-tree callers that pass a verifier at all: the x509 and idemix key managers
+derive it from the identity they are registering, and the `ttx` callers pass `nil`.
+
+**The KVS identity backend cannot compare signer identities.** It does not store the identity
+alongside the signer info, so `GetSignerInfo` there refuses an empty identity but cannot verify that
+the record found under an identity hash belongs to the requested identity. The SQL backend does both.
+This asymmetry is recorded on the KVS method itself; the shared spec in
+`token/services/storage/db/dbtest` holds both backends to everything they *can* both do.
+
+## Why this is sufficient
+
+The empty-identity guards are the least obvious and the most load-bearing, so they are worth stating
+plainly. Identity rows and identity caches are keyed by `Identity.UniqueID()`, and for an empty
+identity that function returns the literal string `<empty>` — **not** a hash. Every empty identity
+therefore collapses onto one row and one cache entry. Without the guard, one caller's audit info,
+token metadata, or signer info is readable by any other empty-identity lookup, and a signer
+registered for one is returned for another. This is why the guard sits on ephemeral registrations
+too, which write nothing to storage but populate the same caches.
+
+The hash-addressed read paths are the second load-bearing case. `GetAuditInfo`, `GetTokenInfo`, and
+`GetSignerInfo` locate a row by `identity_hash`, and `PublicParamsByHash` by `raw_hash` — in each
+case the caller names a value, the store looks up a digest of it, and before these checks nothing
+compared the row it found against what the caller asked for. Comparing them turns a hash-addressed
+read into an authenticated one at the cost of a byte comparison, or one SHA-256 for public
+parameters.
+
+For the remaining classes, the argument is that the check is applied where the information exists.
+Every insert path either just serialized the payload from an in-memory object or just validated it
+in the caller's own scope; re-parsing there would cost time proportional to the payload and learn
+nothing. Every retrieval path hands bytes to a caller that is about to treat them as authentic
+evidence about a specific transaction, and pays one unmarshal to establish that they are.
+
+## If the requirements are not met
+
+The checks are **fail-closed**: a payload that fails one is not stored, and a row that fails one is
+not returned. Retrieval-side failures are logged at ERROR before the error is returned, because they
+indicate storage corruption or out-of-band modification rather than a caller mistake, and nothing
+downstream will surface them a second time.
+
+A caller that skips its own obligations from the Requirements section above is *not* caught by these
+checks — that is what makes them obligations. An unvalidated token request that deserializes and
+carries an action passes the `endorserdb` check and is filed as validated; an acknowledgement with a
+signature that verifies against nothing passes the `tx_ends` check and is filed as an endorsement.
+Both would then be believed by anything reading those stores. The store-level checks narrow what a
+skipped verification can look like; they do not substitute for it.
+
+## Related
+
+* [Endorsement responder security](../services/ttx_responder_security.md) — where token requests
+  arriving over the wire are actually validated.
+* [Storage Service](../services/storage.md) — the schema these checks apply to.
+* [Public Parameters Lifecycle](../public_parameters.md) — why parameters are addressed by hash.
+* [Identity Service](../services/identity.md) — identity registration and the deserializer.
+* [Storage DB Schema Upgradability](../upgradability.md#storage-db-schema-upgradability) — why a
+  persisted digest column is a migration question and not a code change.

docs/services/identity.md

@@ -254,6 +254,35 @@ any other resource, give it a `Close()` method and it will be released automatic
 > **Note:** tests that exercise an anonymous owner wallet should `t.Cleanup(w.Close)`,
 > otherwise each test leaves a provisioning goroutine behind for the rest of the run.
 
+## Identity Registration Checks
+
+Every path that binds something to an identity — audit info, token metadata, signer info, a signer —
+first requires the identity to be **non-empty**, and refuses it outright otherwise. This is not
+input tidying. Identity rows and the provider's caches are keyed by `Identity.UniqueID()`, which maps
+the empty identity to the literal string `<empty>` rather than to a hash, so every empty identity
+would collapse onto one row and one cache entry: one caller's audit info would be readable by any
+other empty-identity lookup, and a signer registered for one would be handed back for another. The
+guard therefore also applies to ephemeral registrations, which write nothing to storage but populate
+the same caches.
+
+Two further checks apply where the information to make them exists:
+
+*   `wallet.Service.RegisterRecipientIdentity` matches the recipient identity against its audit info
+    (`Deserializer.MatchIdentity`) and requires that an **owner verifier can be derived from it**. An
+    identity no verifier can be built from is one whose tokens could never be spent, and both checks
+    route through the same typed-identity deserializer, so the second cannot reject an identity the
+    first accepts.
+*   `token.SignatureService.RegisterSigner` and `RegisterEphemeralSigner` require that **some**
+    verifier — owner, issuer, or auditor — is derivable from the identity a signer is being bound to.
+    They deliberately do not compare a supplied `Verifier` against the identity; `driver.Verifier`
+    exposes only `Verify`, so there is no canonical key to compare.
+
+On the read side, `GetAuditInfo`, `GetTokenInfo`, and (on the SQL backend) `GetSignerInfo` locate a
+row by identity hash and then compare the **stored** identity against the requested one before
+returning or caching anything, so a hash-addressed read cannot silently return another identity's
+data. For the full posture, including the KVS backend's inability to make the last comparison, see
+[**Store Integrity Verification**](../security/store_integrity_verification.md).
+
 ## Identity Types
 
 The Identity Service leverages a wrapper called **TypedIdentity** to support various identity schemes uniformly. 

docs/services/storage.md

@@ -220,6 +220,20 @@ The Storage Service follows a "Finality-Driven" update strategy. While transacti
 The `TokenDB` and `Movements` tables are typically updated only when the **Network Service** confirms that a transaction has reached finality on the ledger. 
 This ensures that the local view of the "Token Landscape" always reflects the ground truth of the distributed ledger.
 
+## Integrity Verification
+
+A store does not persist whatever it is handed. Before a high-value payload is written, and again
+before it is handed back, the store applies a set of structural checks — that a token request is
+anchored to the transaction id it is filed under, that public parameters hash to the hash they are
+addressed by, that a hash-addressed identity row belongs to the identity that was asked for. The
+checks are unconditional and fail-closed, and the ones a store *does not* perform are the caller's
+documented obligation rather than an unstated gap.
+
+Each store method that applies a check names it in a `Verification:` clause in its Godoc, and the
+checks themselves live in `token/services/storage/integrity`. For the full per-asset-class posture —
+including what is deliberately not checked and why — see
+[**Store Integrity Verification**](../security/store_integrity_verification.md).
+
 ## Transaction Recovery Service
 
 The Storage Service includes a **Transaction Recovery Service** that provides the core recovery mechanism for handling pending transactions that may have lost their finality listeners due to node restarts, network interruptions, or other failures.

docs/services/storage/endorserdb.md

@@ -77,6 +77,22 @@ err := store.AppendValidationRecord(
 )
 ```
 
+Unlike TTXDB and AuditDB, which are handed an already-deserialised `*token.Request`, this store
+receives **raw token request bytes taken off the wire**. It therefore checks them before filing them
+as a validated request: `txID`, `tokenRequest`, and `ppHash` must all be non-empty, and
+`tokenRequest` must deserialise at a supported protocol version and carry **at least one action**. A
+validation record asserting that a request was validated is worthless if the request it names carries
+nothing to validate.
+
+The check is not a substitute for validation. The caller
+(`token/services/network/fabric/endorsement/fsc/responder.go`) must still run
+`validator.UnmarshallAndVerifyWithMetadata` and take `ppHash` from its **own** TMS rather than from
+the peer that sent the request. The store's checks narrow what a skipped validation can look like;
+they cannot tell whether the actions are legal. Note also that this format is the bare
+actions-and-signatures encoding, which carries no anchor, so — unlike TTXDB — a record here cannot be
+bound to its transaction id by a structural check. See
+[**Store Integrity Verification**](../../security/store_integrity_verification.md).
+
 ### Querying Validation Records
 
 ```go
@@ -209,4 +225,5 @@ If you're migrating code that previously used ttxdb for validation records:
 ## See Also
 
 - [Storage Services Overview](../storage.md)
-- [TTXDB/AuditDB Documentation](ttxdb.md)
+- [TTXDB/AuditDB Documentation](ttxdb.md)
+- [Store Integrity Verification](../../security/store_integrity_verification.md)

docs/services/storage/ttxdb.md

@@ -38,6 +38,21 @@ It is also possible to append just the transaction records corresponding to a gi
 	}
 ```
 
+### Integrity of appended requests
+
+Appending refuses a request that could not later be checked: an empty transaction id, empty request
+bytes, or an empty public parameters hash are all errors rather than rows. On the way back out,
+`GetTokenRequest` and `GetTokenRequests` deserialize the stored payload and require its **anchor to
+equal the transaction id it is filed under** — callers treat a retrieved request as authentic
+evidence about that transaction, so a payload that is truncated, encoded at an unsupported protocol
+version, or anchored to a different transaction is reported as an error instead of being returned. A
+transaction id with no stored request is still reported as "not found" (`nil`, no error), unchanged.
+
+Note that the `pp_hash` column is the hash of the *public parameters*, not of the request bytes, so it
+is not what makes the retrieved request checkable — the anchor is. See
+[**Store Integrity Verification**](../../security/store_integrity_verification.md) for the reasoning
+and for the endorsement-acknowledgement posture.
+
 ## Payments
 
 The following example shows how to retrieve the total amount of last 10 payments made by a given