Qualified table identidiers#4416
Draft
ilidemi wants to merge 25 commits into
Draft
Conversation
- Table-only destinations (ClickHouse, Elasticsearch) derive physical names via LegacyDotted so legacy dotted destinations keep targeting the same table/index - Add-tables signal path validates additions against the flow's existing tables; structural identifier validation runs even with skip_validation; source dotted collisions rejected (duplicate clone workflow IDs) - Missing entry normalization added: RemoveTablesFromRawTable, QRepHasNewRows, SetupQRepMetadataTables activities and QRepWaitForNewRowsWorkflow - Legacy fallbacks when consuming activity outputs recorded by old releases - Continue-As-New payloads denormalized for rollback safety (continue_as_new.go) - InitialLoadSummary recomposes LegacyDotted names from qrep_runs split columns - CancelTableAddition response denormalized; telemetry reads structs; reset_sequences handles empty-namespace destinations; ScopedEventhub rejects extra dots; QRep watermark validation; UI QRep destination re-parsed at submit; UI removals reuse canonical mappings; delete ParseTableIdentifier (plan 1.3)
- SF golden MERGE SQL byte-identical to pre-refactor generator (fixture captured by running the merge-base generator); dotted per-component quoting + LegacyDotted toast-map keying - BQ raw-name filter pinned to LegacyDotted; CH legacy-normalized dotted destination BuildQuery test - Dotted CTID partitioning unit test (dotted schema/parent/children) - E2e: dotted partitioned-parent ctid QRep, initial-snapshot-only dotted mirror, add-tables union validation rejections, skip-validation structural check, source dotted collision - Snapshot clone workflow ID pinned to the legacy format; pua record.target/source legacy-dotted test; schema-helpers dotted overlap tests; Normalize*/Denormalize* helper round-trips - SF dotted e2e namespace suffixed to avoid cross-run collisions - Review findings doc + progress doc updated
Ports #4414 (shared destinations / N:1 mappings) to QualifiedTable structs: - processTableRemovals keys remainingDestinations/exclusivelyRemovedTables by struct instead of the (cleared) legacy string identifiers - RemoveTablesFromRawTable dedupes destination structs + keeps main's early exit - ClickHouse RemoveTableEntriesFromRawTable chunks IN-lists over LegacyDotted names - validateTableMappingIdentifiers now permits exact-duplicate destinations (N:1 is supported per #4414); dotted-rendering collisions between DIFFERENT destinations are still rejected; e2e duplicate-destination rejection cases dropped accordingly - ui schema.ts resolved to main's version (duplicate-destination check removed)
❌ 2 Tests Failed:
View the top 1 failed test(s) by shortest run time
View the full list of 1 ❄️ flaky test(s)
To view more test analytics, go to the Test Analytics Dashboard |
Contributor
❌ Test FailureAnalysis: The "qualified-table-identifiers" PR deterministically broke SrcTableIdNameMapping population — Test_Dynamic_Mirror_Config_Via_Signals fails identically across all three matrix jobs with an empty map ("should have 1 item(s), but has 0"), which is a real regression rather than a timeout/race/network flake. |
Entry normalization clears the legacy SrcTableIdNameMapping in workflow state; these two assertions were missed in the suite conversion (the PG suite doesn't run on the local dev stack, so first caught by CI).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WIP