Skip to content

Commit c1d8284

Browse files
feat: add Hermes token usage source (#88)
* feat: add Hermes token usage source * test: satisfy all-target clippy checks * fix: address Hermes usage review findings * fix: harden Hermes usage reconciliation * fix Hermes usage edge cases * fix remaining Hermes usage gaps * fix: harden Hermes aggregate accounting * fix: preserve Hermes WAL path identity --------- Co-authored-by: MaterAnime <272887717+MaterAnime@users.noreply.github.com> Co-authored-by: Nico Ritschel <ritschel@gmail.com>
1 parent e930321 commit c1d8284

19 files changed

Lines changed: 2315 additions & 57 deletions

README.md

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# memex
22

3-
Fast local history search for Claude, Codex CLI, Cursor, OpenCode, Pi Coding Agent, OpenClaw, and GitHub Copilot CLI logs. Uses BM-25 and optionally embeds your transcripts locally for hybrid search.
3+
Fast local history search for Claude, Codex CLI, Cursor, OpenCode, Pi Coding Agent, OpenClaw, GitHub Copilot CLI, and Hermes usage records. Uses BM-25 and optionally embeds your transcripts locally for hybrid search.
44

55
Mostly intended for agents to use via skill. The intended workflow is to ask agent about a previous session & then the agent can narrow things down & retrieve history as needed.
66

@@ -210,19 +210,20 @@ Token tracking is disabled by default because it scans and caches local agent lo
210210
token_usage = true
211211
```
212212

213-
Then reconstruct historical token usage from local Claude Code, Codex, Cursor, OpenCode, Pi, OpenClaw, and Copilot logs:
213+
Then reconstruct historical token usage from local Claude Code, Codex, Cursor, OpenCode, Pi, OpenClaw, Copilot, and Hermes records:
214214

215215
```
216216
memex usage
217217
memex usage --source codex --since 2026-07-01
218+
memex usage --source hermes --since 2026-07-01
218219
memex usage --json --events
219220
```
220221

221222
`--cost auto` prefers a provider-stored request cost and otherwise applies the versioned built-in API price catalog. `--cost source` uses only stored costs; `--cost reprice` always applies the catalog. Calculated costs are API-equivalent estimates, not subscription charges. Events with unknown models or prices remain in token totals and are reported as unpriced.
222223

223224
Each source also reports prompt-cache efficiency: the cache hit rate, plus an estimate of cache waste — prompt tokens that were in the previous request's prompt but were re-billed at input rates instead of read from cache, priced at catalog rates and attributed to idle gaps past the cache TTL or model switches where those apply. Waste is estimated per transcript file chain and errs toward undercounting: subagent sidechains, ambiguous dedupe deltas, and prompts that shrink past compaction are not counted.
224225

225-
Local token history is reconstructed usage. It is deliberately kept separate from authoritative subscription quota percentages and reset windows.
226+
Local token history is reconstructed usage. It is deliberately kept separate from authoritative subscription quota percentages and reset windows. Hermes usage is read from `state.db` in the Hermes root and immediate profile directories (`HERMES_PROFILE_ROOTS`, `HERMES_HOME`, or `HERMES_STATE_DIR`, with safe local defaults), opened read-only and WAL-compatible. The `sessions` aggregate is used for legacy databases; newer `session_model_usage` delta rows are emitted by model/task and reconciled against the session aggregate so historical seeded rows count once and positive residuals are retained. Snapshots, backups, arbitrary nested databases, JSON/JSONL transcripts, and auth, config, memory, skills, plugins, and cron paths are excluded. Hermes queries never read message, system-prompt, tool, reasoning-text, or credential tables. Usage output contains counters and metadata only. Any API-equivalent cost estimate is analytical and is not a Hermes subscription quota measurement; source-stored API costs are not quota percentages. Hermes parser-version changes invalidate only Hermes usage cache rows.
226227

227228
When token tracking is enabled, press `Ctrl+T` on the TUI home screen to toggle the 30-day activity chart between session count and token volume. Token activity is loaded lazily and cached when first shown.
228229

@@ -266,7 +267,7 @@ This detects which tools are installed (Claude/Codex/OpenCode/Pi) and presents a
266267
- `--role <user|assistant|tool_use|tool_result>`
267268
- `--tool <tool_name>`
268269
- `--session <session_id>`
269-
- `--source claude|codex|cursor|opencode|pi|openclaw|copilot`
270+
- `--source claude|codex|cursor|opencode|pi|openclaw|copilot|hermes`
270271
- `--since <iso|unix>` / `--until <iso|unix>`
271272
- `--limit <n>`
272273
- `--min-score <float>`

src/cli.rs

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ use std::time::Duration;
3333
#[command(
3434
name = "memex",
3535
version,
36-
about = "Fast local history search for Claude, Codex, Cursor, OpenCode, Pi, OpenClaw, and Copilot",
36+
about = "Fast local history search for Claude, Codex, Cursor, OpenCode, Pi, OpenClaw, Copilot, and Hermes",
3737
after_help = "\
3838
QUICK START:
3939
memex # Browse sessions interactively
@@ -183,7 +183,7 @@ OUTPUT FIELDS (--fields):
183183
/// Filter by session ID
184184
#[arg(long)]
185185
session: Option<String>,
186-
/// Filter by source: claude, codex, cursor, opencode, pi, openclaw, or copilot
186+
/// Filter by source: claude, codex, cursor, opencode, pi, openclaw, copilot, or hermes
187187
#[arg(long)]
188188
source: Option<SourceFilter>,
189189
/// Use semantic (embedding-based) search instead of keyword search
@@ -301,7 +301,7 @@ EXAMPLES:
301301
/// Filter by project (repository grouping)
302302
#[arg(long)]
303303
project: Option<String>,
304-
/// Filter by source: claude, codex, cursor, opencode, pi, openclaw, or copilot
304+
/// Filter by source: claude, codex, cursor, opencode, pi, openclaw, copilot, or hermes
305305
#[arg(long)]
306306
source: Option<SourceFilter>,
307307
/// Only include sessions active on or after this date/timestamp
@@ -336,7 +336,7 @@ EXAMPLES:
336336
memex usage --source codex --since 2026-07-01
337337
memex usage --json")]
338338
Usage {
339-
/// Filter by source: claude, codex, cursor, opencode, pi, openclaw, or copilot
339+
/// Filter by source: claude, codex, cursor, opencode, pi, openclaw, copilot, or hermes
340340
#[arg(long)]
341341
source: Option<SourceFilter>,
342342
/// Only include events on or after this date/timestamp
@@ -452,7 +452,7 @@ enum HerdrCommand {
452452
/// Prefer sessions from this directory (falls back to the global latest)
453453
#[arg(long)]
454454
cwd: Option<PathBuf>,
455-
/// Filter by source: claude, codex, cursor, opencode, pi, openclaw, or copilot
455+
/// Filter by source: claude, codex, cursor, opencode, pi, openclaw, copilot, or hermes
456456
#[arg(long)]
457457
source: Option<SourceFilter>,
458458
/// Path to memex data directory [default: ~/.memex]
@@ -2544,6 +2544,7 @@ fn run_share(session_id: String, title: Option<String>, root: Option<PathBuf>) -
25442544
crate::types::SourceKind::Pi => "pi",
25452545
crate::types::SourceKind::OpenClaw => "openclaw",
25462546
crate::types::SourceKind::Copilot => "copilot",
2547+
crate::types::SourceKind::Hermes => "hermes",
25472548
};
25482549
let source_path = &record.source_path;
25492550

src/ingest.rs

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -743,6 +743,7 @@ pub fn ingest_all(
743743
SourceKind::Copilot => {
744744
parse_copilot_session(task, &tx_record, &tx_update, &next_doc_id, &progress)
745745
}
746+
SourceKind::Hermes => Err(anyhow!("Hermes indexing is not supported")),
746747
};
747748
finish_file_task(task, &progress, &parse_skipped, result)
748749
})
@@ -2926,8 +2927,8 @@ mod tests {
29262927
let (tx_update, rx_update) = unbounded();
29272928
let next_doc_id = AtomicU64::new(1);
29282929
let progress = Arc::new(Progress::new(
2929-
[0, 0, 0, 0, 0, 0, meta.len()],
2930-
[0, 0, 0, 0, 0, 0, 1],
2930+
[0, 0, 0, 0, 0, 0, meta.len(), 0],
2931+
[0, 0, 0, 0, 0, 0, 1, 0],
29312932
false,
29322933
));
29332934

src/progress.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -245,6 +245,7 @@ fn progress_label(source: SourceKind) -> &'static str {
245245
SourceKind::Pi => "pi",
246246
SourceKind::OpenClaw => "openclaw",
247247
SourceKind::Copilot => "copilot",
248+
SourceKind::Hermes => "hermes",
248249
}
249250
}
250251

src/resume.rs

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ pub fn resume_template(config: &UserConfig, source: SourceKind, remote: bool) ->
3030
SourceKind::Pi => config.pi_resume_cmd.clone(),
3131
SourceKind::OpenClaw => return None,
3232
SourceKind::Copilot => config.copilot_resume_cmd.clone(),
33+
SourceKind::Hermes => None,
3334
};
3435
configured.or_else(|| default_resume_template(source.label(), remote))
3536
}

src/sources/audit.rs

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,13 @@ pub fn audit_installed_sources(source: Option<SourceFilter>) -> Result<Vec<Sourc
7272
.map(|file| file.path)
7373
.collect(),
7474
);
75+
push(
76+
SourceKind::Hermes,
77+
super::hermes::discover()
78+
.into_iter()
79+
.map(|file| file.path)
80+
.collect(),
81+
);
7582

7683
groups
7784
.into_iter()
@@ -100,6 +107,12 @@ pub fn audit_files(source: SourceKind, files: &[PathBuf]) -> Result<SourceAudit>
100107
}
101108

102109
fn audit_file(source: SourceKind, path: &Path, audit: &mut SourceAudit) -> Result<()> {
110+
if source == SourceKind::Hermes {
111+
// Hermes usage truth is SQLite aggregate data. Audit must not reinterpret the
112+
// database as JSON, and in particular must not read transcript/message columns.
113+
std::fs::File::open(path)?;
114+
return Ok(());
115+
}
103116
let reader = std::io::BufReader::new(std::fs::File::open(path)?);
104117
for line in reader.lines() {
105118
let line = line?;
@@ -191,6 +204,11 @@ fn record_semantics(source: SourceKind, value: &Value, top_level: &str, audit: &
191204
}
192205
record_content_blocks(value.get("content"), audit);
193206
}
207+
SourceKind::Hermes => {
208+
if value.get("records").and_then(Value::as_array).is_some() {
209+
increment(&mut audit.semantic_types, "records");
210+
}
211+
}
194212
}
195213
}
196214

src/sources/claude.rs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -628,8 +628,10 @@ pub(crate) fn parse_usage_file(path: &Path) -> Result<Vec<UsageEvent>> {
628628
.map(str::to_string),
629629
tokens,
630630
source_cost_usd: value.get("costUSD").and_then(|value| value.as_f64()),
631+
cost_authoritative: false,
631632
dedupe_confidence: if exact_dedupe { "exact" } else { "heuristic" },
632633
conservative_undercount: false,
634+
cache_chain_excluded: false,
633635
sidechain: value
634636
.get("isSidechain")
635637
.and_then(|value| value.as_bool())

src/sources/codex.rs

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1236,9 +1236,11 @@ pub(crate) fn parse_usage_file(
12361236
delta.reasoning,
12371237
),
12381238
source_cost_usd: None,
1239+
cost_authoritative: false,
12391240
dedupe_confidence: "strong",
12401241
conservative_undercount: counter.interleaved
12411242
|| (parent.is_some() && !fork_resolved),
1243+
cache_chain_excluded: false,
12421244
sidechain: false,
12431245
source_order,
12441246
});
@@ -1281,8 +1283,10 @@ pub(crate) fn parse_usage_file(
12811283
tokens.reasoning,
12821284
),
12831285
source_cost_usd: None,
1286+
cost_authoritative: false,
12841287
dedupe_confidence: "strong",
12851288
conservative_undercount: false,
1289+
cache_chain_excluded: false,
12861290
sidechain: false,
12871291
source_order,
12881292
});

src/sources/copilot.rs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -680,8 +680,10 @@ fn extract_usage(
680680
.max(attribute_u64(attributes, "gen_ai.usage.reasoning_tokens")),
681681
),
682682
source_cost_usd: None,
683+
cost_authoritative: false,
683684
dedupe_confidence: "exact",
684685
conservative_undercount: false,
686+
cache_chain_excluded: false,
685687
sidechain: false,
686688
source_order: index,
687689
});

src/sources/cursor.rs

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -488,12 +488,14 @@ fn extract_usage(
488488
}),
489489
tokens: TokenBuckets::disjoint(input, 0, 0, output),
490490
source_cost_usd: None,
491+
cost_authoritative: false,
491492
dedupe_confidence: if table == "cursorDiskKV" {
492493
"exact"
493494
} else {
494495
"strong"
495496
},
496497
conservative_undercount: false,
498+
cache_chain_excluded: false,
497499
sidechain: false,
498500
source_order: 0,
499501
});

0 commit comments

Comments
 (0)