OBSL-1027: SQL AST window functions + UNION-in-CTE#2776
Conversation
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ect DQN via dialect hooks Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… DQNs Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nmatch Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… present (OBSL-1025) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…E (OBSL-1027) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…handler-level test (OBSL-1013) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…to obsl-1027-sql-ast-window-functions-and-cte-union
…core' into obsl-1025-extend-payload-builder-with-metrics-field
…t (OBSL-1027) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The handler discovers with unscoped prefixes, which doesn't surface the test table on databricks/sparkdf shared metastores. The test verifies the handler's connection open/close lifecycle, which is dialect-independent, so one dialect suffices. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…core' into obsl-1025-extend-payload-builder-with-metrics-field
…to obsl-1027-sql-ast-window-functions-and-cte-union
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…core' into obsl-1025-extend-payload-builder-with-metrics-field
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…to obsl-1027-sql-ast-window-functions-and-cte-union
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Add `"scanId": os.environ.get("SODA_SCAN_ID", None)` to build_discovery_payload
- Extract resolve_scan_definition_name helper with precedence: CLI arg > SODA_SCAN_DEFINITION env > default
- Use resolve_scan_definition_name in handle_discover_data_source
- TDD: 5 new unit tests covering scanId env set/unset and all 3 resolution precedence levels
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
for more information, see https://pre-commit.ci
…L-1013) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All CLI dependency resolvers now share one contract: expected/unusable configuration shapes (missing flag, unparseable/invalid config, validation rejection) raise ScanExecutionFailedException carrying the user-facing message — nothing logged at the raise site — while genuinely unexpected failures (e.g. ImportError from a missing plugin) propagate raw so the caller logs them with the traceback, preserving the deliberate traceback-for-data-source-creation asymmetry. run_with_failure_reporting sheds all construction knowledge: it takes the already-constructed reporting channel (soda_cloud) and a zero-arg command, and owns only the Logs lifecycle, the two log-form arms (SEFE clean, everything else with traceback) and failure reporting. The discover wiring resolves the reporting channel first in a small guard (unusable → exit 4, nothing can be reported); the data source and the mandatory scan definition name resolve inside the wrapped command so their failures take the standard mark-with-logs mapping. The local flow adopts the same exception contract with ad-hoc semantics (log → LOG_ERRORS). One nuance: cloud-resolution failure records are no longer captured by the wrapper's Logs — they were never deliverable anyway. ScanExecutionFailedException now extends SodaCoreException (audited: the only 'except SodaCoreException' site wraps contract-verification query execution, which the discover SEFE path never crosses), and SodaCoreException's docstring states its real role as the package-wide base class. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lumn-discovery-in-soda-core' into HEAD
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Both resolvers add YamlParserException to their expected-shape catch tuples: a YAML syntax error and a nonexistent file path — the two most common user mistakes — raise it from the parsing layer and now map to a clean ScanExecutionFailedException message instead of a raw traceback. The tuples are symmetric across the two resolvers. - Root fix in SodaCloud.from_yaml_source: a config missing the top-level 'soda_cloud' key now raises InvalidSodaCloudConfigurationException (same contract as the sibling api_key checks) instead of logging a debug line and crashing on a None dereference. - exit_with_code is annotated NoReturn so bindings after guard except arms (e.g. soda_cloud in the discover wiring) are provably assigned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
send_discovery_results reached into SodaCloud._execute_command from the discovery module; the transport now lives where it belongs: - SodaCoreInsertScanResultsDTO (soda_cloud_dto): TypedDict mirroring the backend SodaCoreInsertScanResultsCommand field set — required keys are the backend's @NotNull/@notempty fields plus the type discriminator, everything else optional via a total=False split (PEP 563-safe __required_keys__). Intended shared shape for all result-inserting flows; profiling adopts it in soda-extensions. - SodaCloud.insert_scan_results(payload) -> bool: owns the _execute_command call, same accepted-bool contract as mark_scan_as_failed. The contract flow keeps its richer send_check_collection_results (response post-processing) — deliberate, converged separately. - build_discovery_payload returns the DTO; send_discovery_results is deleted; the discovery handler calls soda_cloud.insert_scan_results. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lumn-discovery-in-soda-core' into HEAD
…-with-metrics-field' into HEAD
…n tests, wiring complexity - dependencies.py: drop pydantic ValidationError from the resolver catch tuples — it subclasses ValueError in pydantic v2, so the tuple entry was dead; the coverage is now noted in the resolver docstrings. - test_cli_dependencies.py / test_cli_data_source_handlers.py: hoist path construction and mock setup out of pytest.raises blocks so only the call under test can throw; every assertion is kept. - cli.py: extract _resolve_soda_cloud_for_discovery_or_exit and _discover_data_source_locally from the discover wiring to bring its cognitive complexity under the threshold; behavior unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…cated resolvers - FabricSqlDialect.is_system_schema extends (not replaces) the base information_schema exclusion: discovery now includes views, so Fabric was the one dialect where INFORMATION_SCHEMA objects would surface as discoverable datasets. - The local discovery handler no longer catch-logs: failures propagate raw and the local wiring — the single logging site — maps them to LOG_ERRORS, so connection-open failures are no longer mislabeled as query failures and the wiring's generic arm is reachable again. - resolve_scan_definition_name moves to handlers/dependencies.py next to the other resolvers (its tests move to test_cli_dependencies.py). - discovery_payload reads the scan id through EnvConfigHelper, the canonical accessor. - The DTO's type discriminator is Literal["sodaCoreInsertScanResults"]. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lumn-discovery-in-soda-core' into HEAD
run_with_failure_reporting owns a Logs() collector and reports its records on failure, but until now the wrapped command received nothing. A command that constructs its own inner Logs (e.g. via execute_check_collections with logs=None building per-impl Logs) displaced the wrapper's collector, so records emitted downstream never reached the wrapper's failure report. The command now receives the wrapper's own logs, so a command running a check-collection session can thread it through (each impl built with logs=logs) and keep the failure report complete — matching the shared-logs pattern #2779 relies on. Discovery constructs no inner Logs (DiscoveryRun emits via soda_logger, which already lands in the active wrapper collector), so its wiring accepts and ignores the logs argument. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…lumn-discovery-in-soda-core' into HEAD
|
Reviewed. Clean — no findings. The new node follows the existing patterns closely: Two passing notes, neither actionable here:
|
|
No code changes — agreed it's clean. The two boundary notes are real but out of scope here: partition-by-expression ( |
…dow-functions-and-cte-union # Conflicts: # soda-core/src/soda_core/cli/cli.py # soda-core/src/soda_core/cli/handlers/data_source.py # soda-core/src/soda_core/common/soda_cloud.py # soda-postgres/src/soda_postgres/common/data_sources/postgres_data_source.py # soda-postgres/tests/unit/test_postgres_dialect.py # soda-tests/tests/integration/test_discovery.py # soda-tests/tests/unit/test_cli_data_source_handlers.py # soda-tests/tests/unit/test_cli_discover.py # soda-tests/tests/unit/test_discovery_payload.py # soda-tests/tests/unit/test_payload_metrics_field.py
|



PR chain (soda-core)
metrics[]in scan-results payloadSummary
Adds the SQL-AST primitives the v3 numeric profiler needs, unblocking the profiling SQL port (OBSL-1017). Scope was determined by the OBSL-1014 spike.
WINDOW_FUNCTIONAST node (sql_ast.py) —name,args, optionalpartition_by: list[COLUMN], optionalorder_by(reusesORDER_BY_ASC/ORDER_BY_DESC). RendersNAME(args) OVER ([PARTITION BY ...] [ORDER BY ...]).CTE.cte_queryto acceptUNION(coversUNION_ALL);_build_cterenders it viabuild_union_sql.build_expression_sql.Rendered output examples (asserted in tests):
ROW_NUMBER() OVER (ORDER BY "value_" DESC),SUM("amount") OVER (PARTITION BY "category" ORDER BY "id" ASC), CTE bodies containingUNION/UNION ALL.Test Plan
soda-tests/tests/unit→ 993 passed, 2 skipped, 0 failures🤖 Generated with Claude Code