feat(cloud): emit dataset schema metadata when a schema check runs#2782
Draft
m1n0 wants to merge 1 commit into
Draft
feat(cloud): emit dataset schema metadata when a schema check runs#2782m1n0 wants to merge 1 commit into
m1n0 wants to merge 1 commit into
Conversation
Add a top-level metadata[] block to the sodaCoreInsertScanResults payload
carrying each dataset's discovered columns ({columnName, sourceDataType})
whenever a schema check produced a non-empty column set, so Soda Cloud can
refresh the dataset schema from a v4 contract or data standard.
Matches the v3 discover payload shape: raw un-parameterized data type
(SqlDataType.name), primary-key flag omitted. Emitted once per distinct
dataset that ran a schema check, and omitted entirely when no schema check
discovered columns so an empty schema never reaches the wire (the backend
full-reconciles and would otherwise soft-delete every column).
DTL-1807
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
m1n0
force-pushed
the
DTL-1807-refresh-cloud-dataset-metadata
branch
from
July 15, 2026 13:59
3b64f84 to
5a351fe
Compare
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Summary
v4 contracts / data standards now push the discovered dataset schema to Soda Cloud when a schema check runs.
_build_check_collection_results_json_dictgains a top-levelmetadata[]block on thesodaCoreInsertScanResultspayload:{ "datasetQualifiedName": "<datasource>/<prefix…>/<dataset>", "schema": [ { "columnName": "id", "sourceDataType": "integer" } ] }SqlDataType.name, e.g.character varying— notcharacter varying(255));isPrimaryKeyomitted (the v4 schema check doesn't discover PKs; omitting preserves the backend's existing PK flags).datasetQualifiedName= the collection'ssoda_qualified_dataset_name, which resolves to the exact dataset the contract checks registered — so the backend runs its existingsyncDatasetsColumnsreconciliation against the right dataset.Requires (end-to-end)
Backend counterpart: sodadata/soda#12702 — the contract scan handler must ingest this
metadata[]block. This engine PR is inert until that lands. (Ties into DTL-1807.)Known limitation
SqlDataTypelowercases type names; the backend comparessourceDataTypecase-sensitively. Postgres is identical to v3. For warehouses whose catalog returns uppercase types (e.g. SnowflakeTEXT/NUMBER), the first contract scan of a previously-onboarded dataset surfaces a one-time, self-converging "type changed" event. This is pre-existing shared v3/v4 comparison behavior, not introduced here.Test plan
soda-tests/tests/unit/test_dataset_metadata_wire.py— metadata present with the right shape; un-parameterized type; absent with no schema check; absent when the schema check found no columns; one entry per dataset in a combined multi-file session.DTL-1807