Merge pull request #1 from trouze/pypi-v0.1.0

Rewrite as production-ready Python package (v0.1.0 on PyPI)

Author: trouze

.github/workflows/ci.yml

@@ -0,0 +1,40 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v3
+        with:
+          enable-cache: true
+
+      - name: Set up Python ${{ matrix.python-version }}
+        run: uv python install ${{ matrix.python-version }}
+
+      - name: Install project
+        run: uv sync --all-extras --python ${{ matrix.python-version }}
+
+      - name: Lint
+        run: uv run ruff check .
+
+      - name: Typecheck
+        run: uv run mypy src
+
+      - name: Test
+        run: uv run pytest -q --cov=dbt_dag_opt --cov-report=term-missing

.github/workflows/publish.yml

@@ -0,0 +1,34 @@
+name: Publish to PyPI
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  build-and-publish:
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/project/dbt-dag-opt/
+    permissions:
+      id-token: write  # required for PyPI Trusted Publishing (OIDC)
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v3
+        with:
+          enable-cache: true
+
+      - name: Set up Python
+        run: uv python install 3.12
+
+      - name: Build sdist + wheel
+        run: uv build
+
+      - name: Verify built distributions
+        run: uv run --with twine twine check dist/*
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

.gitignore

@@ -152,6 +152,10 @@ venv.bak/
 
 # mypy
 .mypy_cache/
+.ruff_cache/
+
+# Claude Code local state
+.claude/
 .dmypy.json
 dmypy.json
 

CHANGELOG.md

@@ -0,0 +1,30 @@
+# Changelog
+
+All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2026-04-24
+
+Initial PyPI release. Complete rewrite of the pre-release prototype.
+
+### Added
+
+- `dbt-dag-opt analyze` CLI (Typer) with two input modes:
+  - File mode: `--manifest` and `--run-results` point at local dbt artifacts.
+  - Cloud mode: `--account-id`, `--job-id`, optional `--run-id`, and `DBT_CLOUD_TOKEN` env var (or `--token`) pull artifacts from the dbt Cloud Admin API.
+- Output formats: `table` (rich terminal), `json` (valid, `jq`-friendly), `jsonl`.
+- `--top N` to limit results; `--output` to write to a file.
+- Typed exceptions (`ArtifactLoadError`, `DbtCloudAPIError`, `InvalidArtifactError`, `GraphError`).
+- Package ships with `py.typed` (PEP 561).
+- CI matrix across Python 3.10 / 3.11 / 3.12.
+- PyPI publishing via Trusted Publishers (OIDC) on tag push.
+
+### Changed (vs. prototype)
+
+- Replaced per-source recursive DFS + ProcessPoolExecutor with a single iterative DP over topological order. O(V + E) across all sources, no recursion-limit risk, no 20s per-task timeout.
+- Node weights are now attached to the *target* node of each path hop (fixes a bug where parent weights were assigned to outgoing edges).
+- Adjacency list replaces full-edge-list rescan on every DFS step.
+- Output is valid JSON by default (prototype's `longest_paths.json` was a stream of comma-separated fragments opened in append mode — not parseable).
+
+### Notes for PyPI Trusted Publishing
+
+Before the first `v*` tag is pushed, configure PyPI: Project settings → Publishing → Add GitHub publisher with `trouze/dbt-dag-opt` / workflow `publish.yml` / environment `pypi`.

README.md

@@ -1,17 +1,105 @@
 # dbt-dag-opt
-Struggling with long running dbt pipelines? Use this utility to determine the most troublesome paths through your dbt DAG by total execution time. Just because a model is long running, doesn't mean improving it's run time will materially speed up your dbt jobs. Long chained models with comparatively faster runtimes can add up and slow down total pipeline execution time. This utility uses a longest path algorithm to determine your longest running paths through your DAG, starting with each of your sources in dbt.
 
-This package uses [Fire](https://python-fire.readthedocs.io/en/latest/) to run like a CLI. To get started, you can either run using the dbt Cloud Admin API, or pass file paths for your `manifest.json` and `run_results.json` files.
+[![CI](https://github.com/trouze/dbt-dag-opt/actions/workflows/ci.yml/badge.svg)](https://github.com/trouze/dbt-dag-opt/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/dbt-dag-opt.svg)](https://pypi.org/project/dbt-dag-opt/)
+[![Python](https://img.shields.io/pypi/pyversions/dbt-dag-opt.svg)](https://pypi.org/project/dbt-dag-opt/)
+
+**Find the longest-running paths through your dbt DAG — the models that actually make your pipeline slow.**
+
+When you pay for compute by the second (Snowflake, Databricks, Redshift), your dbt job's wall-clock cost is bounded by the *critical path* through the DAG: the longest cumulative chain of model execution times. Optimizing a slow model on a short branch saves you nothing if a longer branch was already the bottleneck. `dbt-dag-opt` tells you which paths to cut first.
+
+## Install
+
+```bash
+pip install dbt-dag-opt
+```
+
+## Quickstart
+
+### From local artifacts
+
+```bash
+dbt-dag-opt analyze \
+  --manifest target/manifest.json \
+  --run-results target/run_results.json \
+  --format table \
+  --top 10
+```
+
+### From dbt Cloud
+
+```bash
+export DBT_CLOUD_TOKEN=dbtu_...
+dbt-dag-opt analyze \
+  --account-id 12345 \
+  --job-id 67890 \
+  --base-url https://cloud.getdbt.com \
+  --format table
+```
+
+Add `--run-id <id>` to pull artifacts from a specific historical run instead of the job's latest.
+
+## Sample output
 
-File path method
 ```
-python3 entrypoint.py --file_method=True --manifest_path='artifacts/manifest.json' --run_results_path='artifacts/run_results.json'
+                       Longest paths by total execution time
+┏━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━┓
+┃ # ┃ Source                    ┃ End of path            ┃ Length ┃ Total time (s) ┃
+┡━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━┩
+│ 1 │ source.demo.raw.orders    │ model.demo.fact_orders │      4 │          35.00 │
+│ 2 │ source.demo.raw.customers │ model.demo.fact_orders │      4 │          32.00 │
+└───┴───────────────────────────┴────────────────────────┴────────┴────────────────┘
 ```
 
-dbt Cloud API
+## CLI reference
+
+```
+dbt-dag-opt analyze [OPTIONS]
+
+  --manifest PATH              Path to manifest.json (file mode)
+  --run-results PATH           Path to run_results.json (file mode)
+  --account-id TEXT            dbt Cloud account id (cloud mode)
+  --job-id TEXT                dbt Cloud job id (cloud mode)
+  --run-id TEXT                dbt Cloud run id; omit for the job's latest run
+  --base-url TEXT              dbt Cloud base URL  [default: https://cloud.getdbt.com]
+  --token TEXT                 dbt Cloud API token  [env: DBT_CLOUD_TOKEN]
+  -f, --format [json|jsonl|table]  Output format  [default: table]
+  -n, --top INTEGER            Show only top N paths (0 = all)  [default: 10]
+  -o, --output PATH            Write output to a file instead of stdout
 ```
-python3 entrypoint.py --account_id='<my_id>' --job_id='<job_id>' --token='<api_token>'
-python3 entrypoint.py --base-url='https://cu288.us1.dbt.com' --account_id='70437463654419' --job_id='70437463655408' --token='dbtu_hayC4-EeNKK-lNbu5xYspNEhbLFeQK1ojfNXAC58J_qr2lRBwA'
+
+### Output formats
+
+- `table` — rich terminal table (default; what you want in a shell).
+- `json` — one object keyed by source: `{source_id: {path, distance, length}}`. Valid JSON, safe to pipe through `jq`.
+- `jsonl` — one JSON object per line. Nice for streaming into a log aggregator.
+
+## How it works
+
+1. **Load** `manifest.json` and `run_results.json` (from disk or dbt Cloud's Admin API).
+2. **Build** a weighted DAG: nodes are `model.*` / `source.*` / `seed.*` / `snapshot.*` ids; each node's weight is its `execution_time` in seconds.
+3. **Compute** the longest path from each source using an iterative DP over topological order (O(V + E)).
+4. **Sort** paths by total distance and surface the heaviest ones.
+
+Distances sum the execution time of every node along the path — that's the warehouse-seconds you'd save by zeroing out that chain.
+
+## What this is / isn't
+
+It **is** a CLI tool that points at the slowest chains in your DAG.
+
+It **isn't** (yet):
+- A scheduler simulator. If your dbt `threads` setting is low, total wall-clock is bounded by parallelism *and* the critical path; v0.2 will surface both. For now, treat the critical-path distance as a lower bound.
+- A cost model. Multiplying distance × your warehouse rate is on you — a `--warehouse-size` flag is planned for v0.3.
+
+## Development
+
+```bash
+uv sync --all-extras
+uv run ruff check .
+uv run mypy src
+uv run pytest
 ```
 
-The utility will save a json file to your working directory that has information on the longest path in your DAG for each starting node (usually sources). It's recommended to use this information to divide and conquer what models you should seek to optimize in order to shorten your pipeline runtimes. 
+## License
+
+Apache 2.0 — see [LICENSE](LICENSE).