Auditor token validation can spuriously fail on recently-audited outputs (async commit-notification race)

[adecaro]

Problem

Auditor-side transaction validation can fail with a spurious error when a
transaction references (as inputs) outputs of a very recently-audited
transaction, because the local read used for audit token lookups is not
guaranteed to reflect a transaction's outputs by the time a dependent
transaction reaches audit validation.

Where this happens

AuditorCheck (token/core/fabtoken/v1/auditor.go,
token/core/zkatdlog/nogh/v1/auditor.go) calls common.RetrieveAuditTokens
(token/core/common/auditor.go), which does a single, non-retrying call to
driver.QueryEngine.ListAuditTokens. That query engine
(token/sdk/vault/vault.go) reads directly from the SQL-backed token store
(token/services/storage/db/sql/common/tokens.go), which returns a hard
error ("token not found for key [...]") the moment a requested token row is
missing — no retry, no backoff.

That SQL row for a transaction's outputs is only written by
tokens.Service.AppendValid, which itself only runs inside the async
finality-listener callback (token/services/ttx/finality/listener.go)
fired once the network layer reports the transaction as final. This is
fully decoupled — in timing — from anything the auditor did when it signed
off on that earlier transaction. A closely-following transaction that spends
or references those same outputs can therefore reach AuditorCheck before
the listener has finished persisting them, causing audit validation to fail
even though the referenced transaction was itself already validated and
signed off by the same auditor.

Existing partial mitigation, and an inconsistency

A different call path — auditor.Service.Audit → Request.AuditRecord
(token/request.go) — uses a different query-engine wrapper
(token.QueryEngine in token/vault.go) that does retry on this exact
condition: it checks whether the owning transaction is Pending and retries
with backoff (3 retries × 3s) before giving up. That wrapper is a different
concrete type from driver.QueryEngine and cannot be substituted into the
driver-level AuditorService.QueryEngine field used by AuditorCheck, so
the retry logic never applies there.

The net effect is an inconsistency: Validate()/AuditorCheck() — which
runs before Audit() in the auditor flow — has no tolerance at all for
this race, while Audit(), which runs after, has a bounded (~9s) grace
window for the same underlying condition.

Additional context: an unused signal

tokens.Service already maintains an in-memory RequestsCache, populated
at audit-approval time (AuditApproveView.Call in
token/services/ttx/auditor.go, via CacheRequest) — i.e. before the
transaction is even committed. That cache entry contains enough information
(TxID, Index, and the token itself) to resolve the exact token IDs that
AuditorCheck looks up. However, neither RetrieveAuditTokens nor either
driver's AuditorService currently has any reference to tokens.Service —
they're constructed with only a driver.QueryEngine — so this signal is
never consulted during audit validation today.

Impact

Auditors can reject (or need to retry) valid, already-approved transactions
purely due to local read timing, not due to any actual problem with the
transaction. This is more likely under load or when transactions chain
quickly (e.g. an output spent shortly after being audited).

Suggestions for investigation (not prescribing a fix)

• Whether AuditorCheck's lookup path should gain the same
  pending-status retry/backoff tolerance that Request.AuditRecord's path
  already has, to remove the inconsistency between the two gates.
• Whether the existing tokens.Service cache could serve as an additional
  fallback for resolving audit tokens that are momentarily missing from the
  DB but already known to be in flight, and if so what plumbing changes
  wiring tokens.Service into AuditorService construction would require
  (currently no such dependency exists in either the fabtoken or zkatdlog
  driver construction chains).

[Effi-S]

Trusting an in memory/ in flight / possibly stale cache to approve transactions not found in the token store seems risky to me.
I'm going to implement option 1 - a backoff (retry)
This, by definition, will result in added latency.

I will add a benchmark that checks this exact scenario to try and measure just how much latency it adds

[Effi-S]

@adecaro,
I want to reconsider keeping it as or significantly reducing the backoff time

In what scenario Is 9 seconds (3 retries of 3 seconds) really necessary?

This is a race condition that warrants a failure in my opinion

The client can retry on his own, or at the very most have 200 millisecond backoffs