Enforce that descriptions live in doc blocks, and document the process (#325)

* First pass at restructuring .md doc files

* Add hooks to pre-commit to enforce .md rules

* Update documentation and claude.md for docs practices

* Cleanup

* Trim comments

* Trim claude.md

* Copilot suggestions

* address copilot review comments

* remove hook exlusion list

* Remove description changes

* Update linter, per copilot comments

* Pass linter, add doc blcoks for toke_balance

Author: ttboogie

.claude/CLAUDE.md

@@ -120,26 +120,14 @@ See `docs/snapshot.md` for the full control flow diagram. For hands-on snapshot
 
 ## Documentation
 
-- **Universal column definitions**: `models/docs/universal.md` — referenced via `{{ doc('column_name') }}` in YAML schema files
-- **Domain docs**: `models/docs/sources/`, `models/docs/snapshots/`, `models/docs/marts/`, `models/docs/intermediate/`
-- Each SQL model has a co-located `.yml` schema file with column descriptions and tests
-
-When adding or modifying models, update both the co-located YAML and any relevant doc blocks in `models/docs/`.
+- **Model or column descriptions:** read `docs/documentation.md` before writing one.
 
 ## Pre-commit Hooks
 
-Pre-commit runs automatically on commit. Run it manually before finishing any task:
+Run before considering a task complete. See `.pre-commit-config.yaml` for the hook list.
 
 ```bash
-pre-commit run --all-files        # Run all hooks on all files
-pre-commit run --files path/to/file.sql   # Run on specific files
+pre-commit run --all-files
 ```
 
-Hooks:
-1. **SQLFluff** — lints and auto-fixes SQL style
-2. **dbt-checkpoint** — enforces:
-   - All model columns in `marts/` must have descriptions in `.yml`
-   - All mart models must have a description
-   - Model tags must be from the approved allowlist (see `.pre-commit-config.yaml`)
-   - All source columns/tables must have descriptions
-3. **Prettier** — formats `.json`/`.yaml`/`.yml` files
+`--files <list>` can report "no files to check" in some environments; `--all-files` is reliable.

.github/pull_request_template.md

@@ -16,6 +16,7 @@ change is, and why it is being made, with enough context for anyone to understan
 
 - [ ] This PR adds tests for the most critical parts of the new functionality or fixes.
 - [ ] I've updated the docs and README with the added features, breaking changes, new instructions on how to use the repository.
+- [ ] If this PR touches column or model descriptions, they live in doc blocks under `models/docs/` rather than inline in the `.yml`, and I read the text of every block I referenced rather than trusting its name (see [docs/documentation.md](../docs/documentation.md)).
 
 ### Release planning
 

.gitignore

@@ -20,3 +20,7 @@ credentials.json
 ### dbt generated files
 package-lock.yml
 .user.yml
+
+# Python bytecode from scripts/
+__pycache__/
+*.py[cod]

.pre-commit-config.yaml

@@ -44,6 +44,18 @@ repos:
 
   - repo: local
     hooks:
+      - id: docs-lint
+        name: Descriptions live in doc blocks, not inline in yml
+        entry: python scripts/docs_lint.py check
+        language: python
+        additional_dependencies: ["pyyaml"]
+        pass_filenames: false
+        # The check reads the whole repo and takes well under a second, so it runs
+        # unconditionally. A `files` filter would skip it on a commit that only
+        # deletes a referenced .md, which is exactly when a dangling doc()
+        # reference appears.
+        always_run: true
+
       - id: cleanup
         name: Cleanup .env.tmp
         entry: ./pre-commit/cleanup.sh

README.md

@@ -20,6 +20,7 @@ If branch is already made, just rename it _before passing the pull request_.
   - [dbt Project Structure](#dbt-project-structure)
     - [Development Folders](#development-folders)
   - [Tests](#tests)
+  - [Documentation](#documentation)
 - [Getting Started](#getting-started)
   - [Download the Repo](#download-the-repo)
   - [Configure dbt](#configure-dbt)
@@ -96,7 +97,7 @@ What not to do in intermediate:
 > _*Note:*_ More information about `intermediate` layer can be found [here](https://docs.getdbt.com/best-practices/how-we-structure/3-intermediate).
 
 3. Marts
-   The marts layer is where users can access final dimensional modeling tables. Each model will be accompanied by a `.yml` file with the same name. The `.yml` file contains the descriptions and tests for the model and its columns.
+   The marts layer is where users can access final dimensional modeling tables. Each model will be accompanied by a `.yml` file with the same name, holding the tests for the model and its columns plus a `{{ doc(...) }}` reference for each description. The description text itself lives in a doc block under `models/docs/`, never inline in the `.yml`. See [Documentation](#documentation).
 
 What to do in marts:
 
@@ -141,7 +142,7 @@ What not to do in snapshots:
 | Staging      | Stages the pre-processed or cleaned data before performing the transformations.               |
 | Intermediate | Contains the transformed and processed data.                                                  |
 | Marts        | Houses the final data models or data marts, which are the end results of the dbt project.     |
-| Docs         | Stores documentation related to your dbt project.                                             |
+| Docs         | Holds the reusable column and model descriptions (doc blocks), mirroring the `models/` layout. |
 | Macros       | Contains reusable SQL code snippets known as macros.                                          |
 | Tests        | Contains defining tests to validate the accuracy and correctness of the data transformations. |
 
@@ -161,6 +162,40 @@ There are three different test types:
 
 <br>
 
+## Documentation
+
+Every model, source table and column carries a description, and those descriptions are published to the dbt docs site on every merge to `master`. There is one rule:
+
+**Descriptions live in doc blocks, never inline in the `.yml`.**
+
+The `.yml` holds a reference; the text lives in a `.md` file under `models/docs/`:
+
+```yaml
+- name: asset_code
+  description: '{{ doc("asset_code") }}'
+```
+
+```markdown
+{% docs asset_code %}
+The 4 or 12 character code representation of the asset on the network.
+{% enddocs %}
+```
+
+That gives each description one home, so when the same column appears in a source, a staging model and a mart, all three point at the same block and the text cannot drift between them.
+
+`models/docs/` mirrors `models/`, so a column on `models/marts/trade_agg.sql` is documented in `models/docs/marts/trade_agg.md`. A definition that **unrelated tables** share, like `asset_code` or `batch_run_date`, goes in `models/docs/universal.md` instead. A column that merely flows from `sources/` through `staging/` to `marts/` is still one table's column and stays in that table's mirror file.
+
+Before writing a description, check whether a block already exists, and read its text rather than trusting its name:
+
+```bash
+grep -rn "{% docs <column_name> %}" models/docs/
+./venv/bin/python scripts/docs_lint.py check     # also runs in pre-commit
+```
+
+> _*Note:*_ The full process, including doc block naming, what the linter deliberately does not check, how to prove a docs change rendered nothing unexpected, and what to check before renaming a block that `stellar-dbt` depends on, is in [docs/documentation.md](./docs/documentation.md)
+
+<br>
+
 
 # Getting Started
 

docs/documentation.md

@@ -0,0 +1,168 @@
+# Documenting models and columns
+
+Every model, source table and column carries a description, published to the dbt docs site on
+every merge to `master` (`.github/workflows/dbt-docs-website.yml`).
+
+- [The rule](#the-rule)
+- [Where a block lives, and what to call it](#where-a-block-lives-and-what-to-call-it)
+- [Adding a column](#adding-a-column)
+- [Running the linter](#running-the-linter)
+- [What the linter does not catch](#what-the-linter-does-not-catch)
+- [Proving a change is docs-neutral](#proving-a-change-is-docs-neutral)
+- [File conventions](#file-conventions)
+- [Relationship to stellar-dbt](#relationship-to-stellar-dbt)
+
+## The rule
+
+**Descriptions live in doc blocks, never inline in yml.** The yml holds a reference:
+
+```yaml
+- name: asset_code
+  description: '{{ doc("asset_code") }}'
+```
+
+and the text lives in a `.md` under `models/docs/`:
+
+```markdown
+{% docs asset_code %}
+The 4 or 12 character code representation of the asset on the network.
+{% enddocs %}
+```
+
+No allowlist, no per-description exception, no way to defer a path, no threshold to tune.
+`scripts/docs_lint.py check` enforces it and runs in pre-commit.
+
+## Where a block lives, and what to call it
+
+`models/docs/` mirrors `models/`: a column on `models/marts/trade_agg.sql` is documented in
+`models/docs/marts/trade_agg.md`. Create the mirror file if it does not exist.
+
+A definition that **unrelated tables** share goes in `models/docs/universal.md` instead, named
+after the bare column (`asset_code`, `batch_run_date`, `closed_at`). A column that merely flows
+`sources/` -> `staging/` -> `marts/` touches several files but is still one table's column, so
+it stays in that table's mirror file.
+
+A definition specific to one model is named `<model>__<column>`, for example
+`int_tvl_trustlines__asset_code` because that column carries non-XLM TVL only. Use the scoped
+form whenever a column's meaning differs from the shared block of the same name, even slightly.
+`amount_raw` (i128 base units) and `amount` (decimals applied) are different concepts and get
+different blocks.
+
+Block names are globally unique across the project and resolve by name, not by file, so moving
+a block between files is always safe.
+
+`check` enforces the root: a definition this repo owns must live somewhere under `models/docs/`.
+Which file it lands in inside that root is convention, not enforced, so the mirror rule above is
+guidance rather than a gate. `docs_lint.py report` shows how many tables use each block if you
+want to check whether something has outgrown its home.
+
+## Adding a column
+
+1. Add the column to the model's `.yml` alongside its tests.
+2. Look for an existing block: `grep -rn "{% docs <column_name> %}" models/docs/`
+3. If one exists, read its text and confirm it describes your column. If it does not, write a
+   new block named `<model>__<column>`.
+4. Otherwise write a block in the model's mirror `.md`.
+5. Run `docs_lint.py check`.
+
+## Running the linter
+
+No warehouse connection or credentials needed; it reads the yml and md files directly.
+
+```bash
+./venv/bin/python scripts/docs_lint.py check     # the gate; also runs in pre-commit
+./venv/bin/python scripts/docs_lint.py report    # diagnostics, never fails a build
+```
+
+`check` requires that the whole value of a `description:` is one `doc()` call naming a block that
+exists, so it fails on inline text, on text wrapped around a reference
+(`see {{ doc("x") }} for detail`), on an empty description, on two references in one value, and
+on a name that is not defined anywhere. It also fails on a yml that will not parse (the rule
+cannot be applied to a file that cannot be read) on two files defining the same block name
+(dbt cannot resolve a duplicate), and on a definition this repo owns that sits outside
+`models/docs/`, which is how a definition drifts somewhere nobody thinks to look.
+
+**Macro and argument descriptions are out of scope.** They document one macro's signature, so
+there is nothing to factor out, and moving them into `models/docs/` would only put a macro's API
+docs further from the macro.
+
+`report` lists orphan blocks, how many tables use each block, columns declared twice under one
+resource, and any column name that resolves to more than one description.
+
+## What the linter does not catch
+
+**A `doc()` that points at the wrong block.** The rule checks that a description *is* a
+reference, not that the reference is *correct*, so both of these pass:
+
+```yaml
+- name: asset_b_type
+  description: '{{ doc("asset_a_type") }}'      # renders "the sold asset", not "the bought asset"
+- name: operation_id
+  description: '{{ doc("transaction_id") }}'    # renders "a unique identifier for this transaction"
+```
+
+Both of those are live in this repo right now, along with nineteen more. They come from copying
+a neighbouring line and cluster on paired columns: `asset_a` / `asset_b`, `read_bytes` /
+`write_bytes`, `batch_id` / `batch_run_date`. Repairing them changes published text, so it is
+tracked separately in #326 and deliberately not part of the change that added this linter.
+
+This is a code review responsibility. When reviewing a description change, read the block that
+is referenced rather than trusting its name. The last section of `docs_lint.py report` helps: a
+column that disagrees with itself inside one table family is nearly always one of these.
+
+**The same column declared twice in one yml.** yaml keeps the last entry, so the first
+description is dropped and the column publishes the second one's text. dbt does not warn.
+`report` lists these; five remain, and `enriched_history_operations_soroban.memo_type` was one
+of them, which is why `memo` had no description at all.
+
+## Proving a change is docs-neutral
+
+Moving blocks between files should not change a single rendered description. Prove it:
+
+```bash
+git stash push --include-untracked   # -u matters: a new untracked .yml or .md
+./venv/bin/python scripts/docs_lint.py snapshot --out /tmp/before.json
+git stash pop
+./venv/bin/python scripts/docs_lint.py snapshot --out /tmp/after.json
+./venv/bin/python scripts/docs_lint.py diff /tmp/before.json /tmp/after.json
+```
+
+A pure relocation must print `0 differences`; a text change must print exactly what you meant
+to change. Paste that output into the PR: it is the cheapest way to show a reviewer that a large
+diff is safe.
+
+The snapshot is trustworthy because the resolver is checked against dbt itself with
+`docs_lint.py validate-manifest`, which needs a `target/manifest.json` from `dbt parse` or
+`dbt docs generate`. It exits non-zero unless `MISMATCHED` is 0, nothing is unresolved, and every
+description it could not compare is one an installed package patched in, which this repo cannot
+see by design.
+
+One thing that looks like it should work and does not: **a `doc()` call inside a doc block.** dbt
+renders block bodies without the `doc` macro in scope and fails the parse with
+`'doc' is undefined`, so a shared definition cannot be wrapped with a local qualifier. Write a
+scoped `<model>__<column>` block with the full text instead.
+## File conventions
+
+- One `.md` per model or source table, mirroring the model's path. Where sibling models each
+  need only a model-level description, one file for the group is fine (see
+  `models/docs/intermediate/trades/int_trade_agg.md`).
+- Open each file with a subject comment: `[comment]: < Trade Aggregations -`. Most files use
+  exactly that form, dangling hyphen included; prefer it in new files.
+- Split `universal.md` into subject files (`asset.md`, `ledger_state.md`, `batch.md`) at around
+  60 to 80 blocks. Below that, one file is easier to search.
+- `models/docs/sources/history_operations.md` holds 125 blocks and is the one file that is
+  genuinely too large. Splitting it is tracked separately.
+
+## Relationship to stellar-dbt
+
+`stellar-dbt` installs this repo as a git package and references roughly 200 doc blocks defined
+here. **Never rename or delete a block** without checking there first: a rename breaks its parse
+and costs a `packages.yml` pin bump plus `dbt deps` to unwind. Moving a block between files is
+safe.
+
+To check before merging something risky, point `stellar-dbt`'s `packages.yml` at your branch and
+run `dbt deps && dbt parse` there. A clean parse confirms every cross-repo reference resolves.
+
+`scripts/docs_lint.py` is meant to be identical in both repos: it reads
+the project name from `dbt_project.yml` rather than hardcoding it, so a change made here is
+copied across verbatim rather than ported by hand.

models/docs/intermediate/int_asset_metadata.md

@@ -21,3 +21,11 @@ The full, human-readable name for the token (e.g., "Circle USD").
 This is part of the standard metadata required for SEP-41 compliance. Consistent naming ensures that the token is handled correctly by explorers and interoperable contracts built to support Soroban's built-in tokens.
 
 {% enddocs %}
+
+{% docs int_asset_metadata %}
+One row per contract_id with the contract's asset_code (coalesced from stg_assets SAC asset_code, then SEP-41 symbol read from contract storage) plus the SEP-41 metadata fields (symbol, name, decimal, admin) read directly from contract instance storage. asset_code is null when neither a SAC asset_code nor a SEP-41 symbol is available — recognized assets are enriched upstream in stg_assets, so a null here means the contract publishes no metadata.
+{% enddocs %}
+
+{% docs int_asset_metadata__asset_code_source %}
+Indicates which source the asset_code came from: 'sac' = asset_code from a Stellar Asset Contract token transfer event, 'metadata' = SEP-41 symbol read from contract storage. Null when no asset_code could be resolved (asset_code is also null in that case).
+{% enddocs %}

models/docs/intermediate/tokens/transfers/int_token_transfer_enrichment.md

@@ -0,0 +1,5 @@
+[comment]: < Token Transfer Enrichment -
+
+{% docs int_token_transfer_enrichment %}
+This table joins token_transfers_raw with history_operations to add operation type and soroban flag.
+{% enddocs %}

models/docs/intermediate/trades/int_trade_agg.md

@@ -0,0 +1,23 @@
+[comment]: < Trade Aggregation Intermediates -
+
+{% docs int_trade_agg_day %}
+Intermediate for the aggregation of daily trade data. This table contains the aggregated metrics of all trades between an asset pair, regardless of whether they are selling or buying.
+{% enddocs %}
+
+{% docs int_trade_agg_month %}
+Intermediate for the aggregation of monthly trade data. This table contains the aggregated metrics of all trades between an asset pair, regardless of whether they are selling or buying.
+{% enddocs %}
+
+{% docs int_trade_agg_week %}
+Intermediate for the aggregation of weekly trade data. This table contains the aggregated metrics of all trades between an asset pair, regardless of whether they are selling or buying.
+{% enddocs %}
+
+{% docs int_trade_agg_year %}
+Intermediate for the aggregation of yearly trade data. This table contains the aggregated metrics of all trades between an asset pair, regardless of whether they are selling or buying.
+{% enddocs %}
+
+{% docs int_trade_agg__day_agg %}
+
+Date from which all metrics are aggregated.
+
+{% enddocs %}

models/docs/intermediate/tvl/int_tvl.md

@@ -0,0 +1,21 @@
+[comment]: < TVL Intermediates -
+
+{% docs int_tvl_accounts %}
+This table aggregates XLM selling liabilities from accounts to calculate TVL
+{% enddocs %}
+
+{% docs int_tvl_trustlines %}
+This table aggregates asset selling liabilities from trustlines to calculate TVL
+{% enddocs %}
+
+{% docs int_tvl_trustlines__account_id %}
+The account id for the tvl
+{% enddocs %}
+
+{% docs int_tvl_trustlines__asset_code %}
+The asset code of the TVL, this table only has non XLM tvl.
+{% enddocs %}
+
+{% docs int_tvl_trustlines__asset_issuer %}
+The asset issuer of the TVL, this table only has non XLM tvl.
+{% enddocs %}