Skip to content

Support non-indexed sourced_from source types in data warehouse ingestion #1359

Description

@ellisandrews-toast

Context

#1273 added support for pure-source types: types that feed sourced_from fields on other types without calling t.index (see #1342, #1343, #1346, #1347, #1356). Events for these types are validated and their sourced fields flow to the destination type's index, but the warehouse gems don't handle them.

Problem

Events for a non-indexed source type never reach the data warehouse:

  1. No warehouse table is defined. Warehouse tables hang entirely off indices: every t.index automatically gets a table (FactoryExtension#new_index in elasticgraph-warehouse/lib/elastic_graph/warehouse/schema_definition/factory_extension.rb), the warehouse_table / exclude_from_warehouse customization API lives on the index object (index_extension.rb), WarehouseTable itself is Data.define(:name, :index), and ResultsExtension#generate_warehouse_config collects tables via each type's own_index_def. A type with no index has no table and no place to declare one.
  2. No operation survives to be dumped. WarehouseDumper#build_jsonl_file_from treats the __self update target (where op.update_target.type == op.event.fetch("type")) as the event's "primary write" (elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda/warehouse_dumper.rb:116). A non-indexed type has no __self update target — its events only produce operations targeting the destination type's index — so every operation is filtered out and no S3 file is written (the dumper skips empty groups).

Note that (2) isn't just a matter of relaxing the filter: no operation carries the full source-event payload (the sourced_from update targets' script params only contain the sourced field values, shaped for the destination document). Warehouse ingestion would need a representation of the raw source event itself.

The warehouse is arguably the one place raw source events should remain visible — the datastore only ever holds the copied fields — so silently dropping them seems wrong.

What's needed

  • Generalize warehouse table generation to cover ingestible non-indexed types (the same "ingestible types" set used for the event envelope enum in Include non-indexed sourced_from source types in the event envelope type enum #1346, exposed as Results#sourced_update_targets_by_source_type_name). The column derivation should port cleanly — WarehouseTable#table_schema already builds from indexing_fields_by_name_in_index, which non-indexed types have — but the table needs to be keyed off the type rather than an index, and the warehouse_table / exclude_from_warehouse API needs a home for types without an index block.
  • Give WarehouseDumper a primary-write signal (and a payload source) that doesn't depend on a __self update target.

Open question

Indexed types get a warehouse table automatically, with exclude_from_warehouse as the opt-out. Should pure-source types follow the same automatic-with-opt-out convention, or should their tables be explicit opt-in?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions