Add skip_record_validation_percents_by_type indexer config for sampled backfill validation

[vermatron]

Add skip_record_validation_percents_by_type indexer config for sampled backfill validation

Why

During large backfills of already-validated data, per-record JSON schema validation is wasted work. Every record walks the full schema (regex, enum, min/max, format, abstract-type discriminators) even though the source has already been validated upstream. Today there's no way to trade that cost for throughput.

This adds a config option that skips per-record validation for a chosen percentage of records, per GraphQL type, while keeping a sampled slice validated as a canary so schema drift still surfaces. It's a sibling of the existing skip_derived_indexing_type_updates backfill knob and follows the same shape.

Design notes:

• skip_record_validation_percents_by_type maps a type name to a percent in [0, 100]. 0 (or an absent key) validates everything, 100 skips everything, and values in between sample. The value is the percentage skipped, so 90 skips 90% and validates 10%.

• The skip decision compares a Zlib.crc32 of the event id (type:id@vversion) against the configured percentage of the CRC32 space. Same event id, same decision, so a retry never flips a record between validated and skipped, even across pods. String#hash won't do here: its seed is per-process, so two pods would disagree. The <= 0 and >= 100 guards keep both endpoints exact, so no float boundary error can make a 0 skip a record or a 100 validate one.

• The event envelope is always validated. Only the per-record schema walk gets sampled.

• Skipping isn't silent. Processor counts skipped records per batch and logs one RecordValidationSkipped line with per-type counts, the same way it logs ElasticGraphIndexingLatencies. Logging per record would drown a 100 percent backfill in log lines, so the count is aggregated per batch.

• Isolation via re-validation, not an error taxonomy: with validation off, malformed data surfaces as an exception while building the event's operations, and there is no bounded list of error types to enumerate. So when validation was skipped, Factory#build rescues anything and then runs the validation it skipped. If the validator faults the record, the caller gets a FailedEventError carrying the validator's own message - the same one it would have gotten on the validated path, PII-sanitized - and the rest of the batch still indexes. If the validator is happy, the error was never about the data (a schema artifact defect, or a bug), so the original exception is re-raised untouched. This is why no dedicated error type is needed for the missing/unknown abstract-type __typename case: Inventor has required: ["__typename"] plus a oneOf whose branches each pin __typename with a const, so a missing one fails required and an unknown one fails every branch.

• ::Kernel.raise is used for the re-raise because Operation::Factory overrides raise to stop the class originating errors instead of returning a BuildResult. Propagating an error that already escaped a collaborator is the opposite case, so the guard is deliberately bypassed, with a comment at the site and a spec that asserts the original class and message. Asserting on both matters: a plain raise exception would substitute the guard's own error and still satisfy a bare raise_error.

• Fixed on the way: build_failed_result could itself raise while building the operations it attaches to a FailedEventError, masking the malformation it was trying to report. An exception in a rescue body isn't caught by that same rescue, so this had to hold before the re-validation path above could route unvalidated records through it. It turns out to be a live bug on the validated path too: Widget requires cost, so a Widget without it fails validation, reaches build_failed_result, and dies with KeyError building the derived WidgetCurrency target, whose id comes from cost.currency. The batch dies with it and the malformation is never reported. It now falls back to no operations and logs FailedEventOperationBuildingFailure; FailedEventError#operations is already documented as sometimes empty for exactly this reason.

• With validation off, one class of failure is still not isolated: malformations that surface only when an operation is serialized for the datastore. Update#to_datastore_bulk is lazy and memoized, so the rollover index suffix and custom routing value computed in Update#metadata are evaluated inside router.bulk, after build has returned - out of reach of any rescue here. Such a batch produces no partial-failure response, so the queue redelivers all of its events and the malformed record fails them again on each retry until it drains to the DLQ. The config documentation says this specifically rather than implying a broader guarantee. Happy to take it on in a follow-up.

The field defaults to {}, so nothing changes unless you set it. Additive and minor-release-safe.

What

Config:

indexer:
  skip_record_validation_percents_by_type:
    Widget: 90     # skip validation for 90% of Widget records, validate 10%
    Component: 100 # skip validation for all Component records

• config.rb: new skip_record_validation_percents_by_type JSON schema property (object, per-type number in [0, 100], additionalProperties: false, default {}); convert_values coerces percents to Float. The description: leads with what the setting does, then the indexing-CPU tradeoff, then what remains unisolated.
• operation/factory.rb: new skip_validation?(type, event) and the CRC32_SPACE_PER_PERCENT constant; build branches on the skip decision into build_success_result or build_success_result_isolating_malformed_records; build_failed_result no longer lets a second failure mask the first. BuildResult gains type_with_skipped_validation.
• processor.rb: aggregate RecordValidationSkipped log per batch when any record was skipped.
• indexer.rb: wire config.skip_record_validation_percents_by_type into the factory.
• record_preparer.rb: unchanged. It carried a RecordPreparer::UnknownTypeError in an earlier revision of this PR; that's gone, and the file no longer appears in the diff.
• RBS signatures updated for all of the above.
• elasticgraph-local config_schema.yaml: regenerated via script/update_config_artifacts.

Verification

• script/run_specs (COVERAGE=1, real Elasticsearch): 5307 examples, 0 failures. elasticgraph-indexer on its own is 264 examples, 0 failures, at 100% line (592/592) and 100% branch (135/135).
• script/type_check (Steep): no type errors.
• script/lint (Standard Ruby): 899 files, no offenses.
• script/spellcheck (codespell): clean.
• script/ci_parts/run_misc_checks: config_schema.yaml is up-to-date, so no artifact drift.
• bundle exec rake schema_artifacts:check: up to date (runtime config only, no artifact changes).
• bundle exec rake site:validate: HTML-Proofer clean over 171 files, 149 runs, 0 failures.
• Generated configuration reference checked by hand: the new field, its text and its examples all appear, and the generated example config still validates against the schema.

New tests:

• config_spec.rb: integer YAML percents coerce to Float (90 to 90.0), and out-of-range percents (100.5, -0.1) are rejected at config load.
• operation/factory_spec.rb: a skipped type builds operations without record validation; non-skipped types still fail on bad records; envelope validation still runs for skipped types; partial sampling (stubbed Zlib.crc32 for both branches); retry stability; the derived-index path under skip; a non-coercible amount_cents and an unknown abstract-type __typename each reported as a FailedEventError carrying the validator's message; an error the validator has no opinion about re-raised with its class and message intact; the validated path still propagating, so the rescue is provably gated; and a malformed event whose operation building also fails still reporting the malformation, with a warn log for the error it discarded.
• processor_spec.rb: a batch with skips logs one RecordValidationSkipped with the right count/counts_by_type; a batch with no skips logs none.

[CLAassistant]

All committers have signed the CLA.

[marcdaniels-toast]

@vermatron I'm digging into this PR. Meanwhile can you merge the latest origin/main into it?

[marcdaniels-toast]

This looks good to me. I'll ask @myronmarston to take a look in case there are bigger picture things I didn't think of or catch.

[myronmarston]

Nice work @vermatron! I left some suggestions.

[myronmarston]

LGTM! Before merging, can you update the PR title and description to reflect where we landed? Currently it refers to skip_record_validation_for which is not the final API. I'll merge this with a squash-and-merge which will use the PR title/description for the commit message so it would be nice if it was accurate.

[vermatron]

LGTM! Before merging, can you update the PR title and description to reflect where we landed? Currently it refers to skip_record_validation_for which is not the final API. I'll merge this with a squash-and-merge which will use the PR title/description for the commit message so it would be nice if it was accurate.

Hi @myronmarston , I have updated the PR description. Thanks for your help with the changes.