Create configuring-state-aware-orchestration Agent Skill - #101
Conversation
…urce info schema tables info
|
@claude Do an adversarial review of this pr |
|
Of note, I use adversarial review against all my stuff all the time. Soooo powerful, and not a judgement |
|
@will-sargent-dbtlabs no worries, I love it! |
|
Doesn't look like Claude took the bait. I'll read this manually monday |
| stg_customers ─┤── fct_order_summary | ||
| stg_products ──┘ | ||
| ``` | ||
| Best candidate for `updates_on: all` — wait for all sources before rebuilding. |
There was a problem hiding this comment.
thought:
I can see this being an issue when there are many upstream refs and one of them is a slowly updating table. For example, maybe stg_products only update every other day in a smaller business. This means we are waiting to build fct_order_summary until someone on product updates product items, which is probably not what we want.
suggestion:
I wonder if there is a way to figure out where this might be a problem - maybe AI can read the source freshness or make an educated guess based on metadata about data update frequency to make a more informed decision around any vs all.
|
|
||
| **If a source is a view:** `loaded_at_field` or `loaded_at_query` is **required** — Fusion treats views as "always fresh" without it. | ||
|
|
||
| **If a source is a table:** `loaded_at_field` or `loaded_at_query` is **recommended** — Fusion can detect freshness from warehouse metadata automatically, but explicit config improves accuracy. |
There was a problem hiding this comment.
issue:
In internal analytics we just removed all of these -- it takes longer and cost money to query a loaded_at_field, whereas the metadata queries are free.
| 1. `list_jobs` → find deploy job "Nightly Marts Refresh" (runs daily at 6am) | ||
| 2. `get_job_details` → extract `execute_steps: ["dbt build -s +fct_orders+"]` | ||
| 3. `get_lineage` for `fct_orders` → returns: `stg_orders`, `stg_customers`, `int_order_enriched`, `fct_orders`, `report_daily_sales` | ||
| 4. Now you know all 5 models run on a daily schedule → recommend `build_after: {count: 1, period: day}` as the baseline for this group |
There was a problem hiding this comment.
question:
if you have a job scheduled at 5am daily and build_after 24 hours and assuming perfect timing:
day 1, 5am job runs.
day 2, 5am job doesnt run (build_after = 23:59). models skip.
Would build_after 23 hours make more sense to ensure it does run every 24 hours?
|
|
||
| | Job Schedule | Suggested Starting `build_after` | Reasoning | | ||
| |-------------|----------------------------------|-----------| | ||
| | Every hour | `{count: 4, period: hour}` | Skip 3 out of 4 runs if data hasn't changed | |
There was a problem hiding this comment.
nitpick:
Skip 3 out of 4 runs if data hasn't changed
In actuality this would be skip 3 out of 4 runs regardless, right?
| | Job Schedule | Suggested Starting `build_after` | Reasoning | | ||
| |-------------|----------------------------------|-----------| | ||
| | Every hour | `{count: 4, period: hour}` | Skip 3 out of 4 runs if data hasn't changed | | ||
| | Every 6 hours | `{count: 12, period: hour}` | Skip one full cycle if data is stale | |
There was a problem hiding this comment.
nitpick:
Skip one full cycle if data is stale
In actuality this would be skip 1 cycle each time, right?
|
This is awesome - would love to see it in action (especially with grabbing jobs schedules and iterating through models/folders. Super cool!
|
3 participants
resolves #88
Description
Key design decisions
updates_on: allto drive cost savingsChecklist
changie newto create a changelog entry