Skip to content

Commit 8ea8da1

Browse files
leifericfclaude
andcommitted
fix(cost): make cost telemetry survive non-Anthropic models
llm-cost-by-model and llm-cost-total returned empty against a fully analyzed db with 358 analyze txes. Three compounding bugs: 1. model-pricing keyed by date-stamped ids Provider responses now carry undated model names ("claude-sonnet-4-6" from the LevelInfinite/Tencent gateway, "glm-4.6" from Z.ai). The pricing table only had "claude-sonnet-4-6-20250514" etc., so lookup missed every response and estimate-cost returned 0.0. Switched to prefix-match against undated keys so both bare and date-stamped ids hit. Also added claude-opus-4-7. 2. :tx/cost-usd asserted only when (pos? cost) With cost=0 for GLM-quota and now-misnamed Claude responses, the attribute was never written. Drop the guard — write 0.0 explicitly so the schema is uniform across txes. 3. cost queries used bare datalog clauses [?tx :tx/cost-usd ?cost] silently excludes any tx without the attr, which (after the bugs above) was every tx. Switch to get-else with 0.0 default. llm-cost-total also gains a :tx/op anchor (analyze or synthesize) so it doesn't pull in import/enrich/seed rows that never had token attributes. Probed both providers directly: neither GLM nor Tencent return cost in usage — they relay Anthropic-format JSON unchanged. The fix is local pricing and local query hygiene, nothing provider-side. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent a7d2964 commit 8ea8da1

5 files changed

Lines changed: 25 additions & 17 deletions

File tree

resources/queries/llm-cost-by-model.edn

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,6 @@
33
:query [:find ?model (sum ?in) (sum ?out) (sum ?cost)
44
:where
55
[?tx :tx/model ?model]
6-
[?tx :tx/input-tokens ?in]
7-
[?tx :tx/output-tokens ?out]
8-
[?tx :tx/cost-usd ?cost]]}
6+
[(get-else $ ?tx :tx/input-tokens 0) ?in]
7+
[(get-else $ ?tx :tx/output-tokens 0) ?out]
8+
[(get-else $ ?tx :tx/cost-usd 0.0) ?cost]]}

resources/queries/llm-cost-total.edn

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22
:description "Total LLM token usage and estimated cost"
33
:query [:find (sum ?in) (sum ?out) (sum ?cost)
44
:where
5-
[?tx :tx/input-tokens ?in]
6-
[?tx :tx/output-tokens ?out]
7-
[?tx :tx/cost-usd ?cost]]}
5+
[?tx :tx/op ?op]
6+
[(contains? #{:analyze :synthesize} ?op)]
7+
[(get-else $ ?tx :tx/input-tokens 0) ?in]
8+
[(get-else $ ?tx :tx/output-tokens 0) ?out]
9+
[(get-else $ ?tx :tx/cost-usd 0.0) ?cost]]}

src/noumenon/analyze.clj

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -315,10 +315,10 @@
315315
:tx/analyzer (or analyzer "noumenon.analyze/0.1.0")
316316
:prov/model-version (or model-version "unknown")
317317
:prov/prompt-hash (or prompt-hash-val "")
318-
:prov/analyzed-at (Date.)}
318+
:prov/analyzed-at (Date.)
319+
:tx/cost-usd cost}
319320
(:input-tokens usage) (assoc :tx/input-tokens (:input-tokens usage))
320-
(:output-tokens usage) (assoc :tx/output-tokens (:output-tokens usage))
321-
(pos? cost) (assoc :tx/cost-usd cost))))
321+
(:output-tokens usage) (assoc :tx/output-tokens (:output-tokens usage)))))
322322

323323
(defn analysis->tx-data
324324
"Convert a parsed analysis map into Datomic tx-data for a file.

src/noumenon/llm.clj

Lines changed: 12 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -85,16 +85,22 @@
8585
;; --- Pricing ---
8686

8787
(def model-pricing
88-
"Per-token pricing in $/1M tokens. Only for direct Anthropic API models.
89-
GLM uses quota-based pricing so is not listed here."
90-
{"claude-sonnet-4-6-20250514" {:input 3.0 :output 15.0}
91-
"claude-haiku-4-5-20251001" {:input 0.80 :output 4.0}
92-
"claude-opus-4-6-20250514" {:input 15.0 :output 75.0}})
88+
"Per-token pricing in $/1M tokens for direct Anthropic API models.
89+
Keys are matched as prefixes against the model id returned by the
90+
provider, so both bare names (claude-sonnet-4-6) and date-stamped
91+
ids (claude-sonnet-4-6-20250514) hit the same entry. GLM and other
92+
quota-priced providers return 0."
93+
{"claude-sonnet-4-6" {:input 3.0 :output 15.0}
94+
"claude-haiku-4-5" {:input 0.80 :output 4.0}
95+
"claude-opus-4-6" {:input 15.0 :output 75.0}
96+
"claude-opus-4-7" {:input 15.0 :output 75.0}})
9397

9498
(defn estimate-cost
9599
"Estimate USD cost for given model and token counts. Returns 0.0 for unknown models."
96100
[model-id input-tokens output-tokens]
97-
(if-let [{:keys [input output]} (model-pricing model-id)]
101+
(if-let [{:keys [input output]} (some (fn [[k v]]
102+
(when (and model-id (str/starts-with? model-id k)) v))
103+
model-pricing)]
98104
(+ (* input-tokens (/ input 1e6))
99105
(* output-tokens (/ output 1e6)))
100106
0.0))

src/noumenon/synthesize.clj

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -547,11 +547,11 @@
547547
:tx/provider (or provider "unknown")
548548
:tx/model-source model-source
549549
:tx/model (or resolved-model model-id "unknown")
550+
:tx/cost-usd cost
550551
:prov/prompt-hash prompt-hash
551552
:prov/analyzed-at (Date.)}
552553
(:input-tokens usage) (assoc :tx/input-tokens (:input-tokens usage))
553-
(:output-tokens usage) (assoc :tx/output-tokens (:output-tokens usage))
554-
(pos? cost) (assoc :tx/cost-usd cost))))
554+
(:output-tokens usage) (assoc :tx/output-tokens (:output-tokens usage)))))
555555

556556
(defn- transact-and-finalize!
557557
"Retract old synthesis, transact new components, derive deps. Returns result map."

0 commit comments

Comments
 (0)