Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ All notable changes to this project will be documented in this file. The format

- `replay` cost overlay: pass `--warehouse-size` (Snowflake XS…6XL) or `--credits-per-hour` (non-Snowflake adapters) to translate wall-clock into dollars. Renders **Run cost**, **Critical-path floor**, **Headroom** (= run − floor; the prize for better parallelization), and **Idle cost** (the $ equivalent of thread-idle warehouse-seconds). Defaults to $2.00/credit (Snowflake Standard On-Demand); override with `--rate-per-credit`. Snowflake's 60-second minimum-billing floor is applied automatically; pass `--no-minimum-billing` to see raw wall-clock × rate.
- New module `dbt_dag_opt.cost` with `CostInputs`, `CostReport`, `compute_cost()`, `credits_per_hour_for()`, and `cost_inputs_from_replay()`. Designed primitive-first so a future `whatif` simulator can call `compute_cost` against simulated schedules and diff the resulting `CostReport`s.
- `scripts/demo.sh` + `tests/fixtures/demo_project/` — narrated end-to-end demo script driving every subcommand against a synthetic 24-model DAG with a shared bottleneck, 4 threads, and ~7.5-min wall-clock. Fixture is regenerable via `tests/fixtures/generate_demo_fixture.py`.

## [0.1.0] - 2026-04-24

Expand Down
16 changes: 16 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,16 @@ It **is** a CLI tool that points at the slowest chains in your DAG, reconstructs
It **isn't** (yet):
- A predictive scheduler simulator. `replay` reconstructs what already happened; it doesn't yet project what would happen under a different `--threads N` or if you sped up a specific model. That "what-if" loop is planned next, and will diff two cost reports to show projected $ savings.

## Demo

An end-to-end walkthrough you can record or run locally:

```bash
./scripts/demo.sh
```

Drives every subcommand (`analyze`, `analyze --show-path`, `replay`, `replay --warehouse-size L/XL`, `--credits-per-hour` for non-Snowflake, JSON + `jq`) against a synthetic 24-model baseball-analytics DAG in `tests/fixtures/demo_project/`. Set `PAUSE=0` to dry-run without narration beats.

## Development

```bash
Expand All @@ -148,6 +158,12 @@ uv run mypy src
uv run pytest
```

Regenerate the demo fixture after editing its topology:

```bash
uv run python tests/fixtures/generate_demo_fixture.py
```

## License

Apache 2.0 — see [LICENSE](LICENSE).
85 changes: 85 additions & 0 deletions scripts/demo.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
#!/usr/bin/env bash
# Demo script for dbt-dag-opt. Designed to be recorded (asciinema / QuickTime).
#
# Runs against a synthetic 24-model dbt project under tests/fixtures/demo_project/
# — a baseball analytics warehouse with 4 threads, ~7.5 min wall-clock, and one
# shared bottleneck (int_game_events) sitting on three of the top longest paths.
#
# Usage:
# ./scripts/demo.sh
#
# Each command is echoed in bold before it runs, with a narration hint above.
# Pause between commands by setting PAUSE=2 (default) or call with PAUSE=0 to
# rush through for a dry-run.

set -euo pipefail

PAUSE="${PAUSE:-2}"
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
MANIFEST="$ROOT/tests/fixtures/demo_project/manifest.json"
RUN_RESULTS="$ROOT/tests/fixtures/demo_project/run_results.json"

bold() { printf "\033[1m%s\033[0m\n" "$*"; }
dim() { printf "\033[2m%s\033[0m\n" "$*"; }
section() {
echo
printf "\033[1;36m▌ %s\033[0m\n" "$*"
echo
}
run() {
bold "\$ $*"
sleep "$PAUSE"
eval "$@"
echo
sleep "$PAUSE"
}

section "1 · Which paths through the DAG are actually slow?"
dim "analyze uses manifest + run_results to compute the critical path — the"
dim "longest cumulative chain of model execution times. That's the bound on"
dim "how fast your pipeline could possibly run."
run "uv run dbt-dag-opt analyze --manifest \"$MANIFEST\" --run-results \"$RUN_RESULTS\" --top 5"

section "2 · The Bottleneck column names the slowest model on each path"
dim "Watch for a model that appears as the bottleneck on MULTIPLE rows — that's"
dim "shared-node leverage. Optimizing it speeds up several paths at once."

section "3 · Drill into the full chain with --show-path"
run "uv run dbt-dag-opt analyze --manifest \"$MANIFEST\" --run-results \"$RUN_RESULTS\" --top 3 --show-path"

section "4 · What actually happened? (replay reconstructs the observed schedule)"
dim "replay reads thread_id + timing from run_results to reconstruct the"
dim "per-thread Gantt, identify the observed critical path, and attribute"
dim "every idle gap to the upstream model a thread was waiting on."
run "uv run dbt-dag-opt replay --manifest \"$MANIFEST\" --run-results \"$RUN_RESULTS\" --top-idle-gaps 5"

section "5 · Put a price on it: --warehouse-size translates wall-clock to dollars"
dim "Four framed numbers:"
dim " • Run cost — what this run billed"
dim " • Critical-path floor — the irreducible cost of your slowest chain"
dim " • Headroom — run − floor; prize for better parallelization"
dim " • Idle cost — \$ equivalent of thread-idle warehouse-seconds"
run "uv run dbt-dag-opt replay --manifest \"$MANIFEST\" --run-results \"$RUN_RESULTS\" --warehouse-size L --top-idle-gaps 3"

section "6 · Change the warehouse, change the bill (same run, XL)"
dim "Doubling warehouse size doubles the rate. Same wall-clock, 2x cost."
run "uv run dbt-dag-opt replay --manifest \"$MANIFEST\" --run-results \"$RUN_RESULTS\" --warehouse-size XL --top-idle-gaps 0"

section "7 · Non-Snowflake adapters: pass --credits-per-hour directly"
dim "Databricks, BigQuery, Redshift — pass the cost/hour your adapter charges."
run "uv run dbt-dag-opt replay --manifest \"$MANIFEST\" --run-results \"$RUN_RESULTS\" --credits-per-hour 12 --rate-per-credit 1.5 --top-idle-gaps 0"

section "8 · Machine-readable: --format json"
dim "Everything in the text output is also in JSON — pipe to jq for dashboards,"
dim "Slack alerts, or CI annotations."
run "uv run dbt-dag-opt replay --manifest \"$MANIFEST\" --run-results \"$RUN_RESULTS\" --warehouse-size L --format json | jq '.cost'"

section "Wrap"
dim "Three takeaways from this run:"
dim " 1. int_game_events is the shared bottleneck on 3 of the top 5 paths."
dim " 2. 5% of the bill is pure parallelism headroom (small — DAG is well-shaped)."
dim " 3. 30% of warehouse-seconds are idle threads — you're overprovisioned"
dim " on thread count for this DAG shape. Consider --threads 2 next run."
echo
bold "pip install dbt-dag-opt"
echo
22 changes: 22 additions & 0 deletions tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@

FIXTURES_DIR = Path(__file__).parent / "fixtures"
DBT_DUGOUT_DIR = FIXTURES_DIR / "dbt_dugout"
DEMO_PROJECT_DIR = FIXTURES_DIR / "demo_project"


@pytest.fixture
Expand Down Expand Up @@ -52,6 +53,27 @@ def dbt_dugout_artifacts(
return DagArtifacts(manifest=manifest, run_results=run_results)


@pytest.fixture
def demo_project_manifest_path() -> Path:
return DEMO_PROJECT_DIR / "manifest.json"


@pytest.fixture
def demo_project_run_results_path() -> Path:
return DEMO_PROJECT_DIR / "run_results.json"


@pytest.fixture
def demo_project_artifacts(
demo_project_manifest_path: Path, demo_project_run_results_path: Path
) -> DagArtifacts:
with demo_project_manifest_path.open() as fh:
manifest = json.load(fh)
with demo_project_run_results_path.open() as fh:
run_results = json.load(fh)
return DagArtifacts(manifest=manifest, run_results=run_results)


def _phase(started: str, completed: str, name: str = "execute") -> dict[str, str]:
return {"name": name, "started_at": started, "completed_at": completed}

Expand Down
Loading
Loading