Skip to content

Codex turns are dropped when last_token_usage components are 0 but total_tokens is set #278

Description

@shwinney-maker

parseCodexToken derives a turn's tokens from three fields of last_token_usage:

let inputTotal = intValue(last["input_tokens"])
let cached     = intValue(last["cached_input_tokens"])
let output     = intValue(last["output_tokens"])
let nonCachedInput = max(0, inputTotal - cached)

Some Codex turns emit a token_count event where every one of those components is 0 while total_tokens is set:

{"timestamp":"2026-09-04T01:29:58.417Z","type":"event_msg","payload":{"type":"token_count",
 "info":{"last_token_usage":{"input_tokens":0,"cached_input_tokens":0,"cache_write_input_tokens":0,
 "output_tokens":0,"reasoning_output_tokens":0,"total_tokens":51293}}}}

The entry is then built with all four token classes at 0, so the turn contributes nothing.

Impact

It is not a format-wide regression — across my 121 rollout files, 24 of 955 token_count events have this shape (~2.5%). But it is not evenly spread, and when a day's Codex activity happens to land entirely in those turns, the provider silently reports zero. That is what happened here: today I had exactly one Codex token_count event, of this shape, worth 51,293 tokens. The menu bar showed codex: 0 with an empty date field, and the 51,293 tokens were dropped.

Environment: codex-cli 0.153.2, PokeTokenBar 2.5.3, macOS 26.

Suggested fix

Fall back to total_tokens when the components sum to zero. This is safe because in the normal case total_tokens already equals exactly what the reader computes — I checked 5 normal events spanning 20K–150K tokens and (input − cached) + cached + output == total_tokens in every one, difference 0. reasoning_output_tokens is already contained in output_tokens, so there is no double count either way.

let nonCachedInput = max(0, inputTotal - cached)
let total = intValue(last["total_tokens"])
// components can all be 0 on some turns while total_tokens is set
let entry: Entry
if nonCachedInput + cached + output == 0, total > 0 {
    entry = Entry(..., input: total, output: 0, cacheWrite: 0, cacheRead: 0)
} else {
    entry = Entry(..., input: nonCachedInput, output: output, cacheWrite: 0, cacheRead: cached)
}

Attributing the fallback to input keeps the day's token total correct; splitting it across classes is not possible since the event carries no breakdown. If skewing the cost estimate is a concern, the alternative is to keep the token total accurate and leave cost at 0 for those turns.

This is the same class of bug as #196 (Cursor reporting zero when the local tokenCount is zero) — a provider whose usage silently vanishes because the reader trusts a breakdown that the source sometimes leaves empty.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions