Commit b66ce5c
feat(snowflake_workspace): parameterized tasks (WITH CONFIG) + multi-instance via assets_by_name
Same three-change pattern as v0.6.5 (procs), applied to tasks:
1. _make_task_asset() gained a config_v parameter. When non-empty, the
asset emits:
EXECUTE TASK <db>.<schema>.<task> WITH CONFIG => '<json>'
where <json> is json.dumps(config_v) with single quotes escaped per
Snowflake's literal rules. Inside the task body,
SYSTEM$GET_TASK_GRAPH_CONFIG() exposes the JSON for SQL / Snowpark
logic. When config_v is empty, the call is the plain EXECUTE TASK
shape (backward-compatible with v0.6.5 and earlier).
2. Factory defined once before the per-task loop so both single-instance
and multi-instance dispatch can call it (mirrors the proc fix).
3. Task-discovery loop now reads ``assets_by_name.<task>`` directly:
── Single-instance (default): one task → one Dagster asset, with an
optional ``config:`` override. Backward-compatible with all
existing configs (no override → no config → identical
EXECUTE TASK <name> shape).
── Multi-instance: one task → N Dagster assets via an ``instances:``
list. Each instance can have its own ``config``, ``asset_name``
(or ``key``), ``description``, ``group_name``, ``deps``, ``kinds``,
``tags``, ``owners``. Lets customers wire per-tenant / per-region
task invocations as N distinct catalog assets — the parent task
definition stays singular in Snowflake while Dagster orchestrates
N parameterized executions.
The observation sensor's task_metadata bucket is populated for each
instance (keyed by the instance's asset_name) so completion events
land on the matching catalog node.
YAML example for multi-instance:
assets_by_name:
REFRESH_REGION:
instances:
- asset_name: refresh_us
config: {region: "US", batch_size: 1000}
kinds: [snowflake, daily]
- asset_name: refresh_eu
config: {region: "EU", batch_size: 500}
deps: [refresh_us]
Both single + multi paths flow through the same _make_task_asset()
factory and get the v0.6.4 numeric-perf-metadata enrichment for free.
_apply_asset_overrides doesn't need touching — config: and instances:
naturally fall through its allow-list (the override is read directly
from self.assets_by_name above).
L1 validation harness scorecard unchanged:
PASS: 562 WARN: 0 FAIL: 41
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>1 parent 0dde8ff commit b66ce5c
3 files changed
Lines changed: 195 additions & 110 deletions
File tree
- dagster_community_components
- integrations/snowflake_workspace
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
21 | 21 | | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
| 24 | + | |
25 | 25 | | |
26 | 26 | | |
27 | 27 | | |
| |||
0 commit comments