Skip to content

Commit 11164e3

Browse files
alex-strukclaude
alex-struk
and
claude
committed
docs(billing): add usage-metering canonical doc + wiki topic (AI-1580)
Document the AI-1580 group usage-metering and spending-cap feature to fit the repo wiki pattern: - docs-md/architecture/USAGE_METERING_AND_BILLING.md: canonical how-it-works / how-to-use reference (data model, rate versions, soft cap check, storage charging, env config, API surface, known limitations). - docs-md/wiki/billing.md: short routing/map topic page with frontmatter and canonical sources; registered in index.md and sources.md. - open-questions.md: record the soft-cap-vs-"atomic"-REQUIREMENTS contradiction and the .env.sample blob-flag name drift. - log.md: ingest entry. Validated with `node scripts/validate-wiki.js` (passes). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 7e72759 commit 11164e3

6 files changed

Lines changed: 259 additions & 0 deletions

File tree

‎docs-md/architecture/USAGE_METERING_AND_BILLING.md‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
# Usage Metering and Billing
2+
3+
Group-scoped usage metering, cost accrual, and optional monthly spending caps. The
4+
system meters every activity that carries a cost (OCR, classification, enrichment,
5+
model/classifier training, and blob storage), records it against the owning group,
6+
rolls it up into per-month summaries, and can block new work before it starts when a
7+
group is about to exceed a configured cap.
8+
9+
The system is generic: costs are driven entirely by a versioned rate file, not by any
10+
document- or workflow-specific logic.
11+
12+
## At a Glance
13+
14+
| Concern | Where it lives |
15+
| --- | --- |
16+
| Shared write path + arg builder | `packages/billing/` |
17+
| Rate seeding, cost estimation, cap check, usage queries, config | `apps/backend-services/src/billing/` |
18+
| Activity interceptor, nightly storage charge, month-end archival | `apps/temporal/src/billing/` |
19+
| Blob storage ledger instrumentation | `apps/backend-services/src/blob-storage/storage-ledger-db.service.ts`, `apps/temporal/src/blob-storage/storage-ledger.ts` |
20+
| Rate definitions (authoritative input) | `apps/backend-services/src/billing/rate_versions.json` |
21+
| Prisma tables | `apps/shared/prisma/schema.prisma` |
22+
| Billing UI | `apps/frontend/src/pages/BillingPage.tsx` |
23+
24+
## Data Model
25+
26+
Six tables (see `schema.prisma` for the source of truth). Money is stored as
27+
`Decimal @db.Decimal(18,8)`, never float.
28+
29+
- **`rate_versions`** — one row per published rate version (`version`, `effective_from`,
30+
`unit_cost_dollars`, `units_per_gb_per_month`, and the estimation assumptions
31+
`max_pages_assumption` / `max_array_items_assumption`).
32+
- **`activity_costs`** — child rows of a rate version: `(rate_version_id, activity_name)`
33+
unique, with `cost_type` (`flat` | `per_page`) and `units`.
34+
- **`usage_events`** — append-only record of every billable event (`activity_completed`,
35+
`workflow_cost`, `model_training`, `blob_storage`). Purged after a retention period.
36+
- **`usage_period_summaries`** — per-group, per-month rollup with a compound unique key
37+
`(group_id, period_year, period_month)` so the summary increment is an atomic upsert.
38+
- **`group_billing_configs`** — the optional monthly cap per group (FK to the group).
39+
- **`group_storage_ledger`** — one row per blob key (`blob_key` unique) tracking
40+
`size_bytes` / `written_at` / `deleted_at`; the input to the nightly storage charge.
41+
42+
Group ownership is derived from the blob key convention `{groupId}/{category}/...`
43+
(see [Blob storage](../wiki/blob-storage.md)); `_shared/` keys are never billed.
44+
45+
## Rate Versions
46+
47+
`rate_versions.json` is the authoritative input. On backend boot,
48+
`RateVersionSeederService` inserts any version present in the file but missing from the
49+
database, and backfills missing `activity_costs` rows (`skipDuplicates`). Existing
50+
versions are **never mutated** — history is immutable, so a rate change means adding a
51+
**new** version with a later `effective_from`.
52+
53+
Each activity cost is either:
54+
55+
- `flat` — a fixed unit count per activity run (e.g. `azureOcr.submit`), or
56+
- `per_page` — units multiplied by the page count reported by the activity.
57+
58+
Dollar cost = `units × unit_cost_dollars`. Storage cost uses
59+
`units_per_gb_per_month` prorated by GB-hours.
60+
61+
> To change pricing: add a new entry to `rate_versions.json` with a bumped `version`
62+
> and a future `effective_from`, then restart the backend to seed it. Do not edit an
63+
> already-seeded version.
64+
65+
## How Costs Are Recorded
66+
67+
All writes go through a single path — `buildUsageEventWriteOps` in `packages/billing`,
68+
executed as one transaction that inserts the `usage_events` row and applies an
69+
`{ increment }` upsert to the month's `usage_period_summaries` row. Period keys are
70+
always UTC.
71+
72+
- **Temporal activities** — the `ActivityBillingInterceptor` records an
73+
`activity_completed` event after each billable activity, looking up the active rate
74+
version per run. Per-page activities report their page count; activities with no cost
75+
entry are skipped.
76+
- **Model / classifier training** — recorded as `model_training` after Azure accepts the
77+
job (`training.service.ts`, `classifier.service.ts`). Billing-write failures here are
78+
logged and do **not** fail the training operation.
79+
- **Blob storage** — the storage ledger tracks inventory; the nightly job converts
80+
stored bytes into `blob_storage` charges (see below).
81+
82+
## Spending Caps (Pre-flight Check)
83+
84+
When a group has a `monthly_cap_dollars` set, cost-incurring entry points estimate the
85+
job cost first and block it if the group's current-period spend plus the estimate would
86+
exceed the cap:
87+
88+
- OCR submission (`ocr.service.ts`) — a blocked document is marked `failed` and a
89+
`402` is raised.
90+
- Model training and classifier training.
91+
92+
Cost estimation (`preflight-cost-estimator.service.ts`) walks the workflow DAG and sums
93+
the longest cost path, applying `max_pages_assumption` / `max_array_items_assumption`
94+
for per-page and fan-out nodes.
95+
96+
> **The cap is a soft cap, not a hard reservation.** The pre-flight check reads current
97+
> spend and compares; it does not reserve budget. Concurrent starts can each pass the
98+
> check and collectively exceed the cap, and jobs that use more pages than the
99+
> assumption can overshoot. See [Known Limitations](#known-limitations).
100+
101+
## Storage Charging
102+
103+
- Every blob write/delete updates `group_storage_ledger` (upsert on `blob_key`, so
104+
re-writing a key refreshes size and clears any tombstone; deletes set `deleted_at`).
105+
- A Temporal **schedule** (`nightlyStorageChargeWorkflow`, cron `5 0 * * *` — 00:05 UTC
106+
daily) charges each group for the previous day's stored bytes as a `blob_storage`
107+
event.
108+
- **Month-end archival** purges `usage_events` older than the retention window
109+
(default 730 days) while preserving the `usage_period_summaries` rollups.
110+
111+
## Configuration
112+
113+
### Environment variables
114+
115+
| Variable | Applies to | Default | Purpose |
116+
| --- | --- | --- | --- |
117+
| `CHARGE_FOR_TEMPORAL_BLOB_TRANSACTION_SEPARATELY` | temporal | `false` | When `true`, Temporal records a per-transaction `blob.write`/`blob.read` charge in addition to activity costs. Recommended `false`: lump blob cost into activity unit costs. |
118+
| `BILLING_TASK_QUEUE` | temporal | `billing-maintenance` | Task queue for the nightly storage charge and archival workflows. |
119+
| `USAGE_EVENT_RETENTION_DAYS` | temporal | `730` | Retention window for `usage_events` before month-end archival purges them. A blank/`0`/invalid value falls back to the default (guards against wiping the audit log). |
120+
121+
> **Known drift:** `.env.sample` currently lists this flag as `CHARGE_FOR_BLOB_TRANSACTION`
122+
> (no value), but the code reads `CHARGE_FOR_TEMPORAL_BLOB_TRANSACTION_SEPARATELY`. Use
123+
> the code name. Tracked in [wiki/open-questions.md](../wiki/open-questions.md).
124+
125+
### Setting a spending cap
126+
127+
`PATCH /api/groups/:groupId/billing-config` — group **ADMIN** (or system admin).
128+
129+
```jsonc
130+
{ "monthly_cap_dollars": 250.00 } // set/raise the cap
131+
{ "monthly_cap_dollars": null } // remove the cap (unlimited)
132+
```
133+
134+
The mutation runs in a transaction that also writes a `billing_cap_update` audit event.
135+
In the UI, admins manage this from the **Billing** page (`SpendingCapView`); the Billing
136+
nav item and page are hidden/redirected for non-admins.
137+
138+
## API
139+
140+
All routes are group-scoped (`minimumRole: ADMIN`) unless noted. Base prefix `/api`.
141+
142+
| Method & path | Access | Purpose |
143+
| --- | --- | --- |
144+
| `GET /usage/summary` | system admin | Cross-group current-period spend. |
145+
| `GET /usage/groups/:groupId/summary` | group admin | Current-period spend, cap, and `usage_percentage`. |
146+
| `GET /usage/groups/:groupId/history` | group admin | Per-month spend history. |
147+
| `GET /usage/groups/:groupId/activity-history?startDate&endDate` | group admin | Per-activity breakdown (optional date window). |
148+
| `GET /usage/groups/:groupId/runs/:workflowExecutionId` | group admin | Cost detail for a single run. |
149+
| `GET /usage/rate-versions` | authenticated | List rate versions. |
150+
| `GET /usage/rate-versions/:versionId/activity-costs` | authenticated | Activity costs for a rate version. |
151+
| `GET /api/groups/:groupId/billing-config` | group-scoped | Read the group's cap. |
152+
| `PATCH /api/groups/:groupId/billing-config` | group admin | Set or clear the cap. |
153+
154+
## Known Limitations
155+
156+
These are current, intentional-or-tracked gaps. Update this section as they are closed;
157+
contradictions with the pre-implementation spec are tracked in
158+
[wiki/open-questions.md](../wiki/open-questions.md).
159+
160+
- **Cap is soft (TOCTOU).** The pre-flight check does not reserve budget, so concurrent
161+
starts can collectively exceed the cap. The `feature-docs` REQUIREMENTS still describe
162+
it as "atomic"; the shipped behavior is a soft cap.
163+
- **No idempotency on retries.** `usage_events` has no idempotency key; a Temporal
164+
activity retry after a completed billing write, or a nightly-job retry, can double-charge.
165+
- **Benchmarks are metered but not cap-checked.** Benchmark workflows can accrue cost
166+
past a cap.
167+
- **Page-assumption undercount.** `max_pages_assumption` (currently `3`) means large
168+
documents pass the pre-flight check well under their real per-page cost.
169+
- **Temporal write-inventory gating.** With `CHARGE_FOR_TEMPORAL_BLOB_TRANSACTION_SEPARATELY`
170+
off (the recommended default), Temporal-written blobs are not added to the storage
171+
ledger, so storage can be under-counted; deletes are always tombstoned regardless.
172+
173+
## Related
174+
175+
- [Blob storage](../wiki/blob-storage.md) — key scheme and provider abstraction the
176+
ledger builds on.
177+
- [Auth and groups](../wiki/auth-and-groups.md) — group scoping and role checks used by
178+
the cap endpoints.
179+
- [Audit](AUDIT.md) — the `billing_cap_update` audit event.
180+
- Pre-implementation context: `feature-docs/20260629185435-usage-metering-billing/`
181+
(REQUIREMENTS, ARCHITECTURE, user stories) — historical, not current-behavior.

‎docs-md/wiki/billing.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
---
2+
status: active
3+
updated: 2026-07-23
4+
canonical_sources:
5+
- docs-md/architecture/USAGE_METERING_AND_BILLING.md
6+
- apps/backend-services/src/billing/
7+
- apps/temporal/src/billing/
8+
- packages/billing/
9+
- apps/backend-services/src/billing/rate_versions.json
10+
do_not_duplicate:
11+
- Rate version values and units
12+
- Full API endpoint list
13+
- Prisma table definitions
14+
- Cost estimation algorithm
15+
---
16+
17+
# Billing and Usage Metering
18+
19+
Group-scoped metering of billable activities (OCR, classification, enrichment,
20+
training, blob storage), per-month cost rollups, and optional monthly spending caps
21+
enforced before work starts. Rates come from a versioned rate file, not per-workload
22+
logic. Full behavior, configuration, and API live in the canonical doc
23+
`docs-md/architecture/USAGE_METERING_AND_BILLING.md`.
24+
25+
## Source Map
26+
27+
- Canonical doc: `docs-md/architecture/USAGE_METERING_AND_BILLING.md`.
28+
- Shared write path and arg builder: `packages/billing/`.
29+
- Backend billing (seeder, cost estimation, cap check, usage queries, config):
30+
`apps/backend-services/src/billing/`; authoritative rates in that folder's
31+
`rate_versions.json`.
32+
- Temporal billing (activity interceptor, nightly storage charge, month-end archival):
33+
`apps/temporal/src/billing/`.
34+
- Blob inventory feeding storage charges: `storage-ledger-db.service.ts` (backend) and
35+
`blob-storage/storage-ledger.ts` (temporal).
36+
- Schema (six `Decimal` money tables): `apps/shared/prisma/schema.prisma`.
37+
- UI: `apps/frontend/src/pages/BillingPage.tsx`.
38+
39+
## Design Notes
40+
41+
- One shared write path: insert a `usage_events` row and atomically `{ increment }` the
42+
`usage_period_summaries` rollup in a single UTC-keyed transaction.
43+
- Rate versions are immutable; a price change is a new version, seeded on backend boot.
44+
- The spending cap is a **soft** pre-flight check, not a budget reservation — see the
45+
canonical doc's Known Limitations and [open-questions](open-questions.md).
46+
47+
## Related Topics
48+
49+
- [Blob storage](blob-storage.md): key scheme and provider abstraction the ledger uses.
50+
- [Auth and groups](auth-and-groups.md): group scoping and role checks for cap endpoints.
51+
- [Graph workflows](graph-workflows.md): the activities the interceptor meters.
52+
- [System overview](system-overview.md): backend vs Temporal vs frontend boundaries.
53+
54+
## Common Drift Risks
55+
56+
- `rate_versions.json` values drift from documented examples; treat the JSON as
57+
authoritative and never restate specific units in the wiki.
58+
- The `feature-docs` REQUIREMENTS describe an "atomic" cap; the shipped behavior is a
59+
soft cap — do not let planning docs be read as current behavior.
60+
- `.env.sample` names the blob-transaction flag differently from the code
61+
(`CHARGE_FOR_BLOB_TRANSACTION` vs `CHARGE_FOR_TEMPORAL_BLOB_TRANSACTION_SEPARATELY`).

‎docs-md/wiki/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ This wiki is a compact map of the Document Intelligence Platform. Use it to find
1313
- [HITL](hitl.md): review sessions, locking, queue behavior, and correction tracking.
1414
- [Auth and groups](auth-and-groups.md): Keycloak/API key auth, group scoping, and authorization docs.
1515
- [Deployment and ops](deployment-and-ops.md): local/dev/OpenShift deployment, monitoring, load testing, and operational runbooks.
16+
- [Billing](billing.md): usage metering, cost accrual, and group spending caps.
1617

1718
## Maintenance Pages
1819

‎docs-md/wiki/log.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,3 +82,9 @@ Use grep-friendly headings: `## [YYYY-MM-DD] operation | Title` where operation
8282
- Added new docs for previously-undocumented merged subsystems: `workflows/TEMPORAL_PAYLOAD_FOOTPRINT.md` (gzip codec + OCR payload refs) and `extraction/OCR_RESULT_VIEWS.md` (AI-1445); routed both from `graph-workflows.md` / `extraction.md` and registered the codec package in `sources.md`.
8383
- Expanded `architecture/AUDIT.md` (group, tables, `document_deleted`, `system_bootstrap` domains; hybrid benchmark audit; `recordEvent(events, tx?)`) and reconciled the stale Findings-vs-Summary contradiction in `TRANSACTION_AND_AUDIT_AUDIT.md`.
8484
- Fixed `benchmarking/LOAD_TESTING.md` HA row: backend `backend-services/pvc.yml` (RWX blob PVC) was removed; blob storage is object storage via `BLOB_STORAGE_PROVIDER`.
85+
86+
## [2026-07-23] ingest | Add Billing topic and canonical usage-metering doc (AI-1580)
87+
88+
- Added `docs-md/architecture/USAGE_METERING_AND_BILLING.md` as the canonical how-it-works/how-to-use doc for the AI-1580 usage-metering + spending-cap feature (data model, rate versions, cap check, storage charging, env config, API, known limitations).
89+
- Added `billing.md` topic page routing to it and the `packages/billing/` + backend/temporal billing source areas; linked from `index.md` and registered in `sources.md` (routing row, stable-docs entry, code-adjacent sources).
90+
- Recorded the soft-cap-vs-"atomic"-REQUIREMENTS contradiction and the `.env.sample` blob-flag name drift in `open-questions.md`.

‎docs-md/wiki/open-questions.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,11 @@ Use this page for contradictions, drift candidates, and ownership gaps that shou
88
- ~~Which docs from `docs-md/` should eventually be published through the generated `docs/` site, and which should remain repo-only?~~ Partially resolved: selected guides (benchmarking, authentication, integrations) are published via `docs/_pages/`; implementation docs and the repo wiki remain in `docs-md/` (wiki HTML is generated at deploy, not committed). Revisit when adding new public pages.
99
- ~~Should the wiki validator become a CI check after the team has used it for a few PRs?~~ Resolved: `.github/workflows/wiki-check.yml` runs `npm run docs:wiki:check` on PRs to `main` and `develop`.
1010

11+
## Billing (AI-1580, in review)
12+
13+
- **Cap semantics contradict the spec.** `feature-docs/20260629185435-usage-metering-billing/REQUIREMENTS.md` describes the spending cap as "atomic" (no two concurrent starts can both pass), but the shipped `preflight-cap-check.service.ts` is a read-only soft check with no budget reservation. `docs-md/architecture/USAGE_METERING_AND_BILLING.md` documents the soft-cap behavior; the REQUIREMENTS wording should be reconciled (or the reservation implemented).
14+
- **`.env.sample` blob-flag name drift.** `.env.sample` lists `CHARGE_FOR_BLOB_TRANSACTION`, but the temporal code reads `CHARGE_FOR_TEMPORAL_BLOB_TRANSACTION_SEPARATELY`. Setting the sample name has no effect; align the sample to the code name.
15+
1116
## Documentation Ownership
1217

1318
- **Root `README.md`**: local setup, prerequisites, development commands, project tree.

‎docs-md/wiki/sources.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ This registry names the places the wiki should route readers to before summarizi
1515
| `docs-md/architecture/HITL_ARCHITECTURE.md` | [HITL](hitl.md) |
1616
| `docs-md/auth/AUTHENTICATION.md`, `docs-md/auth/GROUP_RESOURCE_AUTHORIZATION.md`, `docs-md/groups/` | [Auth and groups](auth-and-groups.md) |
1717
| `docs-md/operations/`, load testing, monitoring, CI workflows | [Deployment and ops](deployment-and-ops.md) |
18+
| `docs-md/architecture/USAGE_METERING_AND_BILLING.md` | [Billing](billing.md) |
1819

1920
Maintenance and registry pages: [Sources](sources.md), [Open questions](open-questions.md), [Log](log.md).
2021

@@ -51,6 +52,7 @@ Maintenance and registry pages: [Sources](sources.md), [Open questions](open-que
5152
- Load testing: `docs-md/benchmarking/LOAD_TESTING.md` — wiki: [Deployment and ops](deployment-and-ops.md)
5253
- Monitoring and alerting: `docs-md/monitoring/LOCAL_MONITORING_STACK.md`, `docs-md/monitoring/ALERTING.md` — wiki: [Deployment and ops](deployment-and-ops.md)
5354
- OpenShift deployment: `docs-md/operations/` — wiki: [Deployment and ops](deployment-and-ops.md)
55+
- Usage metering and billing: `docs-md/architecture/USAGE_METERING_AND_BILLING.md` — wiki: [Billing](billing.md)
5456

5557
## Code-Adjacent Sources
5658

@@ -62,4 +64,7 @@ Maintenance and registry pages: [Sources](sources.md), [Open questions](open-que
6264
- HITL backend: `apps/backend-services/src/hitl/`
6365
- Group backend: `apps/backend-services/src/group/`
6466
- Auth backend: `apps/backend-services/src/auth/`
67+
- Billing backend: `apps/backend-services/src/billing/`
68+
- Billing Temporal (interceptor, storage charge, archival): `apps/temporal/src/billing/`
69+
- Shared billing package: `packages/billing/`
6570
- Frontend feature areas: `apps/frontend/src/features/`, `apps/frontend/src/pages/`

0 commit comments

Comments
 (0)