Skip to content

feat(cloud): emit dataset schema metadata when a schema check runs#2782

Draft
m1n0 wants to merge 1 commit into
mainfrom
DTL-1807-refresh-cloud-dataset-metadata
Draft

feat(cloud): emit dataset schema metadata when a schema check runs#2782
m1n0 wants to merge 1 commit into
mainfrom
DTL-1807-refresh-cloud-dataset-metadata

Conversation

@m1n0

@m1n0 m1n0 commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Summary

v4 contracts / data standards now push the discovered dataset schema to Soda Cloud when a schema check runs. _build_check_collection_results_json_dict gains a top-level metadata[] block on the sodaCoreInsertScanResults payload:

{ "datasetQualifiedName": "<datasource>/<prefix…>/<dataset>",
  "schema": [ { "columnName": "id", "sourceDataType": "integer" } ] }
  • Reuses the v3 discover payload shape: raw, un-parameterized data type (SqlDataType.name, e.g. character varying — not character varying(255)); isPrimaryKey omitted (the v4 schema check doesn't discover PKs; omitting preserves the backend's existing PK flags).
  • datasetQualifiedName = the collection's soda_qualified_dataset_name, which resolves to the exact dataset the contract checks registered — so the backend runs its existing syncDatasetsColumns reconciliation against the right dataset.
  • One entry per distinct dataset that ran a schema check; omitted entirely when no schema check discovered columns (an empty schema is never sent — the backend full-reconciles and would otherwise soft-delete every column).

Requires (end-to-end)

Backend counterpart: sodadata/soda#12702 — the contract scan handler must ingest this metadata[] block. This engine PR is inert until that lands. (Ties into DTL-1807.)

Known limitation

SqlDataType lowercases type names; the backend compares sourceDataType case-sensitively. Postgres is identical to v3. For warehouses whose catalog returns uppercase types (e.g. Snowflake TEXT/NUMBER), the first contract scan of a previously-onboarded dataset surfaces a one-time, self-converging "type changed" event. This is pre-existing shared v3/v4 comparison behavior, not introduced here.

Test plan

  • soda-tests/tests/unit/test_dataset_metadata_wire.py — metadata present with the right shape; un-parameterized type; absent with no schema check; absent when the schema check found no columns; one entry per dataset in a combined multi-file session.
  • Full unit suite: 918 passed, 2 skipped (no regressions in the surrounding wire tests).

DTL-1807

@m1n0 m1n0 self-assigned this Jul 7, 2026
Add a top-level metadata[] block to the sodaCoreInsertScanResults payload
carrying each dataset's discovered columns ({columnName, sourceDataType})
whenever a schema check produced a non-empty column set, so Soda Cloud can
refresh the dataset schema from a v4 contract or data standard.

Matches the v3 discover payload shape: raw un-parameterized data type
(SqlDataType.name), primary-key flag omitted. Emitted once per distinct
dataset that ran a schema check, and omitted entirely when no schema check
discovered columns so an empty schema never reaches the wire (the backend
full-reconciles and would otherwise soft-delete every column).

DTL-1807

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@m1n0
m1n0 force-pushed the DTL-1807-refresh-cloud-dataset-metadata branch from 3b64f84 to 5a351fe Compare July 15, 2026 13:59
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant