Skip to content

Add skill + exporter for dbt Cloud job completion DAG - #110

Draft
patkearns10 wants to merge 3 commits into
mainfrom
feature/skill-export-job-dag
Draft

Add skill + exporter for dbt Cloud job completion DAG#110
patkearns10 wants to merge 3 commits into
mainfrom
feature/skill-export-job-dag

Conversation

@patkearns10

Copy link
Copy Markdown

Description

This PR adds a new dbt-extras Agent Skill that documents how to build a job-level DAG from dbt Cloud Admin API data—specifically job_completion_trigger_condition (run when another job completes)—which is separate from model lineage. It also ships a small Python exporter and a sample Mermaid file so the diagram can be opened in [Mermaid Live](https://mermaid.live/) or other renderers.

Motivation

Orchestration between jobs (including cross-project completion triggers) is easy to miss when looking only at model DAGs or schedules. The skill standardizes how agents gather jobs via dbtp ([dbt Rust SDK](https://github.com/dbt-labs/dbt-rust-sdk)), extract edges, and present results as JSON plus a styled flowchart LR diagram.

What’s included

  • mapping-dbt-cloud-job-dag skill (SKILL.md): workflow for listing jobs, resolving project names, fetching full job definitions, building edges, sanitizing labels, and emitting Mermaid with ELK-friendly %%{init}%%classDef / class styling, solid same-project links, and labeled dashed cross-project links.
  • scripts/export_job_dag.py: optional CLI to generate a .mmd from dbtp (env: DBT_TOKENDBT_ACCOUNT_IDDBT_HOST as needed); supports -cross-project-label-renderer, and DBTP_PATH.
  • job-dag-export.mmd: checked-in example output (regenerate with the script for your projects).
  • Bump dbt-extras plugin version to 1.2.0 to reflect the new skill.

How to verify

  1. Install dbtp per [dbt-rust-sdk](https://github.com/dbt-labs/dbt-rust-sdk) and configure DBT_* env vars.

  2. Run (adjust project IDs / paths as needed):

    python3 skills/dbt-extras/skills/mapping-dbt-cloud-job-dag/scripts/export_job_dag.py
    --project-id <upstream_project_id> --project-id <downstream_project_id>
    -o /tmp/job-dag.mmd

  3. Paste /tmp/job-dag.mmd into Mermaid Live and confirm subgraph colors, three completion edges, and the cross-project label render as expected.

Notes

  • Edges reflect job completion triggers only; schedule/webhook ordering is out of scope (called out in the skill).
  • job-dag-export.mmd is illustrative; consumers can delete it or replace it with a neutral fixture if the repo prefers not to ship account-flavored samples.

Checklist

@patkearns10
patkearns10 requested a review from a team as a code owner May 1, 2026 06:36
@patkearns10

patkearns10 commented May 4, 2026

Copy link
Copy Markdown
Author

Added to this:

  • Progress logging — Writes interim status to stderr ([job-dag-export] prefix): scope summary, project/job counts after listing, milestones during bulk jobs get, connected-only before/after counts, render summary. Helps ~5+ minute account exports feel observable instead of silent.
  • -quiet — Suppresses progress lines; output path on stderr at the end is unchanged.
  • JSON snapshots — On each successful fetch-mode run, writes a .json next to the .mmd (same basename) with full jobs get payloads, project_names, capture metadata, and options used for the fetch. Snapshot is written after edge endpoints are resolved and before -connected-only, so the same file supports toggling connected-only offline.
  • -no-snapshot — Skips writing the JSON when disk size or sensitivity matters.
  • -from-snapshot <path> — Rebuilds edges from the snapshot and renders with -renderer-cross-project-label-connected-only, and o only—no dbtp or DBT_* required. Mutually exclusive with -account-project-id, and -expand-downstream.
  • Docs — SKILL.md updated for stderr progress, -quiet, and the snapshot / -from-snapshot workflow.

@patkearns10

Copy link
Copy Markdown
Author
mode_1_whole_account_all_jobs ---- mode_2_whole_account_connected_only ---- mode_3_all_jobs_for_select_projects ---- mode_4_listed_projects_connected_only ---- mode_5_focus_one_project_and_downstream_connections

@b-per

b-per commented May 4, 2026

Copy link
Copy Markdown
Contributor

The thing is dbtp is not public today.

@b-per
b-per marked this pull request as draft May 4, 2026 11:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants