Generalize indexing schema versions

[jwils]

Why

The shared indexing pipeline required json_schema_version, even though other ingestion formats may have no JSON schemas or version numbers. This PR introduces an optional, format-neutral schema_version and keeps version resolution inside each ingestion adapter.

What

Adapters return a normalized event alongside the record preparer. Operations, latency logs, and warehouse partitions use the schema version actually selected for that event. The JSON adapter accepts both version keys, gives schema_version precedence, and rejects invalid values such as false. These rules apply equally to direct processor calls and decoded payloads.

Decoders identify their format with ingestion_format. The JSON decoder stamps json, and untagged events remain compatible with existing JSON callers. A schema version no longer doubles as a format discriminator, so tagged formats route independently of adapter order and version presence. An explicitly different format is rejected even when JSON is the only installed adapter.

The schema version is optional

Each adapter decides what an absent version means. JSON selects the latest available schema; an unavailable requested version selects the closest available schema. Both validation and record preparation use that selected version.

JSON events with missing versions still enter a versioned warehouse partition after resolution. Formats without versions use the fixed unversioned S3 key segment. Publishers should keep sending explicit JSON schema versions and indexers should retain historical artifacts: replaying unversioned JSON after a schema change can behave differently.

Backward compatibility

• Publishers can keep sending json_schema_version; JSON schema artifacts retain their existing version const.
• Direct legacy events are normalized before downstream processing, preserving their versions in latency logs and warehouse partitions.
• Test factories accept __schema_version, legacy __json_schema_version, or neither. Generic-key precedence matches ingestion, and invalid values remain available for validation.
• Latency and warehouse logs retain the deprecated json_schema_version alias. Both names report the selected version; fallback logs retain their existing requested and selected JSON-version fields.
• The JSON ingestion upgrade guide covers the full stack's runtime dependencies, regenerated artifacts, Lambda decoder settings, and moved matcher and preparer APIs.

Review fixes

Version normalization now has one owner: the JSON adapter. This fixes conflicting-key precedence between decoded and direct events, false versions being treated as absent, and legacy versions disappearing downstream. Processor failures are correlated through normalized operation events, so datastore errors still identify all operations for the failed event.

Regression coverage exercises legacy, versionless, and fallback events through the warehouse pipeline; multiple adapters in both orders; selected-schema field preparation; and failures after normalization.

Risk

This changes the adapter result contract introduced earlier in the stack: successful results now require event:. Downstream logs and warehouse partitions report the selected schema version instead of an unavailable requested version. Optional JSON versions still tie interpretation to the indexer's installed schema; the upgrade guide documents that tradeoff.

Testing

No manual deployment testing. Automated regressions cover the ingestion boundaries and the full repository suite validates the combined stack.

Stack

Current PR is marked with ->.

• #1301 Extract an ingestion adapter seam inside elasticgraph-indexer
• #1302 Move JSON ingestion into elasticgraph-json_ingestion via an indexer extension
• #1220 Add configurable indexing event decoder
• #1351 Extract indexing field metadata behind a format-neutral value object
• -> #1284 Generalize indexing schema versions

[jwils]

Pushed d3ff605, which changes the design in one substantive way and fixes five defects. I rewrote the PR description to match. Summary of the delta for anyone who already reviewed:

schema_version is now optional. The earlier revision required it and failed an event that omitted it. That forced every ingestion format to have an integer version, which protobuf does not: it achieves compatibility through field numbers, which #1304 already pins in a sidecar artifact. The key is now optional at every stage, and each adapter decides what a missing version means. The JSON adapter uses the latest available version and still validates the event against it, so a malformed event still fails early enough to be useful.

The breaking changes are gone. The earlier revision broke three things that I had not called out:

• An existing generated project's shared_factories.rb defines __json_schema_version, so Converters.upsert_event_for raised KeyError.
• An event carrying only json_schema_version failed validation, which affects in-process callers of Indexer#processor.process, not just the SQS path.
• Three operator-facing log fields were renamed, which breaks dashboards silently.

All three now accept the legacy name, and the logs emit both the new key and the deprecated alias.

Defects fixed. The README and the decoder interface both promised a default to the latest version that the earlier revision had removed; the behaviour is back and both texts are now format neutral. This PR had also added a duplicate decoder section to the JSON gem README that named a class which does not exist. The indexer extension memoized into a name its own super call assigns. Version selection re-sorted the available versions on every event.

One limitation stays open. handles_event? cannot tell one format from another, so two live formats still need a real discriminator. I describe the plan in the description; it belongs in the proto runtime PR rather than here.