You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#1273 added support for pure-source types: types that feed sourced_from fields on other types without calling t.index (see #1342, #1343, #1346, #1347, #1356). Events for these types are validated and their sourced fields flow to the destination type's index, but the warehouse gems don't handle them.
Problem
Events for a non-indexed source type never reach the data warehouse:
No warehouse table is defined. Warehouse tables hang entirely off indices: every t.index automatically gets a table (FactoryExtension#new_index in elasticgraph-warehouse/lib/elastic_graph/warehouse/schema_definition/factory_extension.rb), the warehouse_table / exclude_from_warehouse customization API lives on the index object (index_extension.rb), WarehouseTable itself is Data.define(:name, :index), and ResultsExtension#generate_warehouse_config collects tables via each type's own_index_def. A type with no index has no table and no place to declare one.
No operation survives to be dumped.WarehouseDumper#build_jsonl_file_from treats the __self update target (where op.update_target.type == op.event.fetch("type")) as the event's "primary write" (elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda/warehouse_dumper.rb:116). A non-indexed type has no __self update target — its events only produce operations targeting the destination type's index — so every operation is filtered out and no S3 file is written (the dumper skips empty groups).
Note that (2) isn't just a matter of relaxing the filter: no operation carries the full source-event payload (the sourced_from update targets' script params only contain the sourced field values, shaped for the destination document). Warehouse ingestion would need a representation of the raw source event itself.
The warehouse is arguably the one place raw source events should remain visible — the datastore only ever holds the copied fields — so silently dropping them seems wrong.
What's needed
Generalize warehouse table generation to cover ingestible non-indexed types (the same "ingestible types" set used for the event envelope enum in Include non-indexed sourced_from source types in the event envelope type enum #1346, exposed as Results#sourced_update_targets_by_source_type_name). The column derivation should port cleanly — WarehouseTable#table_schema already builds from indexing_fields_by_name_in_index, which non-indexed types have — but the table needs to be keyed off the type rather than an index, and the warehouse_table / exclude_from_warehouse API needs a home for types without an index block.
Give WarehouseDumper a primary-write signal (and a payload source) that doesn't depend on a __self update target.
Open question
Indexed types get a warehouse table automatically, with exclude_from_warehouse as the opt-out. Should pure-source types follow the same automatic-with-opt-out convention, or should their tables be explicit opt-in?
Context
#1273 added support for pure-source types: types that feed
sourced_fromfields on other types without callingt.index(see #1342, #1343, #1346, #1347, #1356). Events for these types are validated and their sourced fields flow to the destination type's index, but the warehouse gems don't handle them.Problem
Events for a non-indexed source type never reach the data warehouse:
t.indexautomatically gets a table (FactoryExtension#new_indexinelasticgraph-warehouse/lib/elastic_graph/warehouse/schema_definition/factory_extension.rb), thewarehouse_table/exclude_from_warehousecustomization API lives on the index object (index_extension.rb),WarehouseTableitself isData.define(:name, :index), andResultsExtension#generate_warehouse_configcollects tables via each type'sown_index_def. A type with no index has no table and no place to declare one.WarehouseDumper#build_jsonl_file_fromtreats the__selfupdate target (whereop.update_target.type == op.event.fetch("type")) as the event's "primary write" (elasticgraph-warehouse_lambda/lib/elastic_graph/warehouse_lambda/warehouse_dumper.rb:116). A non-indexed type has no__selfupdate target — its events only produce operations targeting the destination type's index — so every operation is filtered out and no S3 file is written (the dumper skips empty groups).Note that (2) isn't just a matter of relaxing the filter: no operation carries the full source-event payload (the
sourced_fromupdate targets' script params only contain the sourced field values, shaped for the destination document). Warehouse ingestion would need a representation of the raw source event itself.The warehouse is arguably the one place raw source events should remain visible — the datastore only ever holds the copied fields — so silently dropping them seems wrong.
What's needed
Results#sourced_update_targets_by_source_type_name). The column derivation should port cleanly —WarehouseTable#table_schemaalready builds fromindexing_fields_by_name_in_index, which non-indexed types have — but the table needs to be keyed off the type rather than an index, and thewarehouse_table/exclude_from_warehouseAPI needs a home for types without an index block.WarehouseDumpera primary-write signal (and a payload source) that doesn't depend on a__selfupdate target.Open question
Indexed types get a warehouse table automatically, with
exclude_from_warehouseas the opt-out. Should pure-source types follow the same automatic-with-opt-out convention, or should their tables be explicit opt-in?