Split out of #217 (comprehensive metrics), which is now closed with the core suite delivered. This is a post-v0.9.0 fast-follow.
What
A counter of tokens processed by successfully-dispatched requests — the "unit of value" for LLM workloads (cost/throughput dashboards).
- Metric:
async_tokens_total (counter), labeled by pool plus direction="input"|"output" (and the usual queue_id/queue_name where applicable).
Approach
In the worker success path (2xx from the IGW), best-effort parse the OpenAI usage object from the response body (prompt_tokens, completion_tokens) and increment the counter. No-op when usage is absent or the body isn't parseable.
Caveats
- Introduces response-body parsing; the processor is otherwise payload-agnostic, so this is best-effort and only works for OpenAI-style responses.
- Streaming responses only carry
usage in the final chunk (and only when stream_options.include_usage is set); async/batch traffic is typically non-streaming, so the common path is simple.
Split out of #217 (comprehensive metrics), which is now closed with the core suite delivered. This is a post-v0.9.0 fast-follow.
What
A counter of tokens processed by successfully-dispatched requests — the "unit of value" for LLM workloads (cost/throughput dashboards).
async_tokens_total(counter), labeled by pool plusdirection="input"|"output"(and the usualqueue_id/queue_namewhere applicable).Approach
In the worker success path (2xx from the IGW), best-effort parse the OpenAI
usageobject from the response body (prompt_tokens,completion_tokens) and increment the counter. No-op whenusageis absent or the body isn't parseable.Caveats
usagein the final chunk (and only whenstream_options.include_usageis set); async/batch traffic is typically non-streaming, so the common path is simple.