|
| 1 | +# CLAUDE.md — Project Context for dbtp |
| 2 | + |
| 3 | +## What is this? |
| 4 | + |
| 5 | +`dbtp` is a Rust CLI for the dbt Cloud Platform APIs. It covers three API surfaces: |
| 6 | + |
| 7 | +1. **Admin API** (REST v2/v3) — accounts, projects, environments, jobs, runs, artifacts |
| 8 | +2. **Discovery API** (GraphQL) — models, sources, exposures, macros, seeds, snapshots, tests, semantic models, lineage |
| 9 | +3. **Semantic Layer API** (GraphQL) — metrics, saved queries, dimension values, metric queries |
| 10 | + |
| 11 | +The CLI follows hyperscaler conventions (`aws`, `az`, `gcloud`): flat `resource verb` hierarchy, consistent CRUD verbs, global output formatting, config profiles, auto-pagination, and envelope unwrapping. |
| 12 | + |
| 13 | +## Repository layout |
| 14 | + |
| 15 | +``` |
| 16 | +dbtp/ |
| 17 | +├── Cargo.toml # Rust project manifest (edition 2021, Apache-2.0) |
| 18 | +├── src/ |
| 19 | +│ ├── main.rs # Entry point: parse CLI, load config, build clients, dispatch |
| 20 | +│ ├── api/ # API layer — request/response logic per API surface |
| 21 | +│ │ ├── mod.rs |
| 22 | +│ │ ├── types.rs # Shared API types (envelope, pagination) |
| 23 | +│ │ ├── admin/ # REST Admin API (v2 + v3) |
| 24 | +│ │ │ ├── mod.rs # Compact formatters, re-exports |
| 25 | +│ │ │ ├── accounts.rs |
| 26 | +│ │ │ ├── projects.rs |
| 27 | +│ │ │ ├── environments.rs |
| 28 | +│ │ │ ├── jobs.rs |
| 29 | +│ │ │ ├── runs.rs |
| 30 | +│ │ │ └── artifacts.rs |
| 31 | +│ │ ├── discovery/ # GraphQL Discovery API |
| 32 | +│ │ │ ├── mod.rs # GraphQL execution helpers, require_environment_id |
| 33 | +│ │ │ ├── models.rs # list, parents, children, health, performance |
| 34 | +│ │ │ ├── lineage.rs |
| 35 | +│ │ │ ├── sources.rs |
| 36 | +│ │ │ ├── exposures.rs |
| 37 | +│ │ │ ├── macros.rs |
| 38 | +│ │ │ └── resource_details.rs # Generic detail fetcher for any Discovery node type |
| 39 | +│ │ └── semantic_layer/ # GraphQL Semantic Layer API |
| 40 | +│ │ ├── mod.rs # SL URL construction, re-exports |
| 41 | +│ │ ├── metrics.rs # list, dimensions, entities, measures, granularities |
| 42 | +│ │ ├── queries.rs # execute_query, compile_sql (Arrow IPC decoding) |
| 43 | +│ │ └── types.rs # GroupByInput, OrderByInput, WhereInput, MetricInput |
| 44 | +│ ├── cli/ # CLI layer — clap definitions and command dispatch |
| 45 | +│ │ ├── mod.rs # Cli struct, GlobalOpts, Commands enum (clap derive) |
| 46 | +│ │ ├── output.rs # Output formatting: table (tabled), json, yaml, compact |
| 47 | +│ │ └── commands/ # One file per top-level command |
| 48 | +│ │ ├── mod.rs # exec() dispatcher matching Commands enum |
| 49 | +│ │ ├── configure.rs # Interactive profile setup |
| 50 | +│ │ ├── accounts.rs |
| 51 | +│ │ ├── projects.rs |
| 52 | +│ │ ├── environments.rs |
| 53 | +│ │ ├── jobs.rs # Includes trigger, trigger-from-failure |
| 54 | +│ │ ├── runs.rs # Includes cancel, retry, wait (polling), errors |
| 55 | +│ │ ├── artifacts.rs |
| 56 | +│ │ ├── models.rs # Includes parents, children, health, performance |
| 57 | +│ │ ├── lineage.rs |
| 58 | +│ │ ├── sources.rs |
| 59 | +│ │ ├── exposures.rs |
| 60 | +│ │ ├── macros.rs |
| 61 | +│ │ ├── seeds.rs |
| 62 | +│ │ ├── snapshots.rs |
| 63 | +│ │ ├── tests.rs |
| 64 | +│ │ ├── semantic_models.rs |
| 65 | +│ │ ├── metrics.rs # Includes query, sql, dimensions, entities, etc. |
| 66 | +│ │ ├── saved_queries.rs |
| 67 | +│ │ └── dimension_values.rs |
| 68 | +│ └── core/ # Shared infrastructure |
| 69 | +│ ├── mod.rs |
| 70 | +│ ├── config.rs # Profile loading/saving, env var fallback, config.toml |
| 71 | +│ ├── error.rs # DbtpError enum (Http, Config, GraphQL, Api, Io, Json, Arrow) |
| 72 | +│ ├── rest_client.rs # REST client: v2/v3 URL routing, auth, pagination, envelope unwrapping |
| 73 | +│ └── graphql_client.rs # GraphQL client for Discovery + Semantic Layer |
| 74 | +├── openapi/ # Reference OpenAPI specs (not used at build time) |
| 75 | +│ ├── openapi-v2.yaml |
| 76 | +│ └── openapi-v3.yaml |
| 77 | +├── dbt-mcp/ # Separate project: Python MCP server for dbt |
| 78 | +├── dbtc/ # Separate project: Python dbt Cloud API wrapper |
| 79 | +└── .github/ |
| 80 | + └── CODEOWNERS |
| 81 | +``` |
| 82 | + |
| 83 | +## Key architecture decisions |
| 84 | + |
| 85 | +- **Three-layer separation**: `core/` (config, clients, errors) → `api/` (request/response per API) → `cli/` (clap args, dispatch, output formatting). Commands in `cli/commands/` call into `api/` and return `serde_json::Value`; the dispatcher in `cli/commands/mod.rs` formats the result. |
| 86 | +- **Everything is `serde_json::Value`**: Rather than strongly-typed response structs, commands return `Value` and the output layer formats generically. This keeps the code lean and makes adding new fields/endpoints trivial. |
| 87 | +- **REST v2 vs v3**: The `RestClient` exposes `get_v2`/`get_v3`/etc. methods. v2 uses `Token` auth, v3 uses `Bearer` auth. Each `api/admin/*.rs` module knows which version its endpoints use. Users never see the version split. |
| 88 | +- **GraphQL clients**: `GraphqlClient` handles both Discovery API (at `{host}/graphql`) and Semantic Layer (at a different URL constructed by `semantic_layer::semantic_layer_url`). Discovery uses environment_id as a query variable; Semantic Layer encodes it in the URL path. |
| 89 | +- **Auto-pagination**: `RestClient::paginate_v2` / `paginate_v3` loop through offset-based pages (100 per request) until `total_count` is reached or `--limit` is satisfied. |
| 90 | +- **Envelope unwrapping**: dbt Cloud REST APIs return `{ data, status, extra }`. The client strips this automatically via `unwrap_envelope()`. |
| 91 | +- **Compact mode**: A `compact` output format provides single-line JSON and uses `compact_*` helpers in `api/admin/mod.rs` to reduce verbose API responses to essential fields. |
| 92 | + |
| 93 | +## Building and running |
| 94 | + |
| 95 | +```bash |
| 96 | +cargo build # Debug build |
| 97 | +cargo build --release # Optimized build (strip + LTO) |
| 98 | +cargo run -- projects list # Run directly |
| 99 | +``` |
| 100 | + |
| 101 | +The release profile enables `strip = true`, `lto = true`, `codegen-units = 1` for a small, fast binary. |
| 102 | + |
| 103 | +## Configuration |
| 104 | + |
| 105 | +Config lives at `~/.config/dbtp/config.toml` (macOS) via the `directories` crate's `ProjectDirs::from("com", "dbt-labs", "dbtp")`. |
| 106 | + |
| 107 | +Precedence: CLI flags > env vars (`DBTP_TOKEN`, `DBTP_HOST`, `DBTP_ACCOUNT_ID`, `DBTP_ENVIRONMENT_ID`) > profile config > defaults. |
| 108 | + |
| 109 | +## Key dependencies |
| 110 | + |
| 111 | +| Crate | Purpose | |
| 112 | +|---|---| |
| 113 | +| `clap` (derive) | CLI framework with subcommands and env var integration | |
| 114 | +| `reqwest` (rustls-tls) | Async HTTP client | |
| 115 | +| `tokio` | Async runtime | |
| 116 | +| `serde` / `serde_json` / `serde_yaml` | Serialization | |
| 117 | +| `tabled` | Table output rendering | |
| 118 | +| `toml` | Config file parsing | |
| 119 | +| `thiserror` | Error type derivation | |
| 120 | +| `directories` | XDG-compliant config path resolution | |
| 121 | +| `indicatif` | Progress bars | |
| 122 | +| `chrono` | Timestamp handling | |
| 123 | +| `arrow` (ipc) | Decoding Semantic Layer query results (Arrow IPC format) | |
| 124 | +| `base64` | Decoding base64-encoded Arrow payloads | |
| 125 | + |
| 126 | +## Sibling projects in this repo |
| 127 | + |
| 128 | +- **`dbt-mcp/`** — Python MCP (Model Context Protocol) server for dbt. Separate project with its own dependencies and entry point. Not part of the Rust binary. |
| 129 | +- **`dbtc/`** — Python dbt Cloud API client library and CLI. Separate project. Not part of the Rust binary. |
| 130 | +- **`openapi/`** — Reference OpenAPI v2/v3 specs from dbt Cloud. Used as a reference when writing API modules, not consumed at build time. |
| 131 | + |
| 132 | +## Adding a new command |
| 133 | + |
| 134 | +1. Add the API module in `src/api/{surface}/` (e.g. `src/api/admin/connections.rs`). |
| 135 | +2. Add the command module in `src/cli/commands/` (e.g. `connections.rs`) with `*Args`, `*Command` enum, and `exec()`. |
| 136 | +3. Register the command in `src/cli/mod.rs` (add variant to `Commands` enum) and `src/cli/commands/mod.rs` (add to `exec()` match and module imports). |
| 137 | +4. The output layer handles formatting automatically — just return `serde_json::Value` from your `exec()`. |
| 138 | + |
| 139 | +## Releasing |
| 140 | + |
| 141 | +Push a `v*` tag to trigger the release workflow at `.github/workflows/release.yml`. It cross-compiles for macOS (arm64 + x86_64) and Linux (x86_64 + arm64), then creates a GitHub Release with tarballs and SHA-256 checksums. The `install.sh` script at the repo root fetches the latest release for the user's platform. |
| 142 | + |
| 143 | +```bash |
| 144 | +git tag v0.1.0 && git push --tags |
| 145 | +``` |
| 146 | + |
| 147 | +## Testing |
| 148 | + |
| 149 | +No test suite currently. The CLI is tested manually against live dbt Cloud instances. Set `DBTP_TOKEN` and `DBTP_ACCOUNT_ID` in your environment and run commands directly. |
0 commit comments