Skip to content

Commit 77ccb65

Browse files
manojmallickmanojmallick
committed
chore(release): v8.23.0 — changelog, version sync, docs, benchmarks
- CHANGELOG 8.23.0 'Agent Economy I': sigmap budget spend ledger + get_budget MCP tool (#508, PR #509) - version sync via sync-versions.mjs; benchmarks re-run — metrics unchanged from v8.22 (hit@5 82.2%, honest lift 1.59x, hard split 33.3% vs grep 53.3%, token reduction 96.8%); benchmark id advanced to sigmap-v8.23-main via gen-benchmark-latest + sync-metrics - docs-vp: cli.md budget section (80 commands), config.md sessionBudgetTokens/contextTtlDays rows, mcp.md get_budget row, roadmap v8.23.0 entry + stats line fix (82.2%/1.59x/21 tools), snapshot labels bumped to v8.23 - docs/index.html softwareVersion 8.23.0; llms regenerated - suite: 123 passed, 0 failed
1 parent 52640ad commit 77ccb65

31 files changed

Lines changed: 150 additions & 92 deletions

‎CHANGELOG.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,18 @@ Format: [Semantic Versioning](https://semver.org/)
1010

1111
---
1212

13+
## [8.23.0] — 2026-07-28
14+
15+
Minor release — **"Agent Economy I" (v8.23, F1)**: SigMap's token savings become queryable *during* a session. A spend ledger over the existing gain log, an optional budget threshold, and context-freshness age — as a CLI command and the 21st MCP tool.
16+
17+
### Added
18+
- **`sigmap budget` + MCP `get_budget` (#508, PR #509):** new `src/tracking/budget.js` — `budgetStatus()` sums the session's gain-log entries (estimated SigMap-emitted tokens: spent/baseline/saved, op count), computes remaining/pct against an optional budget, and reports generated-context age with a stale flag. Session identity: `SIGMAP_SESSION` env override, else UTC day bucket; `recordUsage` now stamps every entry with `session` (legacy entries match day buckets by timestamp prefix). New opt-in config keys `sessionBudgetTokens` and `contextTtlDays` (both default `null`). CLI: `sigmap budget [--json] [--session <key>] [--budget <tokens>]`. The MCP tool advises degrade-gracefully tactics (terse encoding, `squeeze`, summarize-then-drop) at ≥80% budget. 7 new integration tests (129 test files); tool-count guards advanced 20→21.
19+
20+
### Changed
21+
- **Scope honesty by design:** the ledger counts tokens **SigMap emitted** (chars/4, labeled `estimated-tokens` on every surface) — not the host chat's total spend, which a CLI cannot see. Context freshness replaces the originally-planned prompt-cache "injection TTL", which was unverifiable and got cut in the plan audit.
22+
23+
---
24+
1325
## [8.22.0] — 2026-07-28
1426

1527
Minor release — **"Hard Corpus" (v8.22)**: the benchmark corpus gains a no-leakage hard split with a deterministic leakage gate, and per-repo-size buckets stop tiny repos from flattering the average. The headline retrieval number gets harder — and honest.

‎CONTRIBUTORS.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,9 @@ To ensure proper attribution:
3535

3636
We welcome contributions! See [Contributing](./docs/CONTRIBUTING.md) for guidelines.
3737

38+
### Recent Contributors (v8.23.0)
39+
- **@manojmallick** — feat(tracking,mcp): `sigmap budget` session spend ledger (estimated SigMap-emitted tokens, optional budget, context-age TTL) + `get_budget` MCP tool (21st); `SIGMAP_SESSION` identity; opt-in `sessionBudgetTokens`/`contextTtlDays` (#508, PR #509)
40+
3841
### Recent Contributors (v8.22.0)
3942
- **@manojmallick** — feat(eval,benchmarks): hard-split corpus + deterministic leakage gate (`src/eval/corpus.js`, `scripts/validate-task-corpus.mjs`) + repo-size buckets in `benchmark:honest`; 15 leak-free hard tasks; measured the hard-split ceiling (33.3% vs grep 53.3%) (#505, PR #506)
4043
- **@octo-patch** — feat(eval): MiniMax LLM-ablation provider — `MINIMAX_API_KEY`, OpenAI-compatible endpoint, default MiniMax-M3, pricing entry + tests (PR #504)

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -122,8 +122,8 @@ Ask → Rank → Context → Validate → Judge → Learn
122122

123123
<!--SM:benchmarkBlock-->
124124
```
125-
Benchmark : sigmap-v8.22-main (21 repositories, including R language)
126-
Date : 2026-07-27
125+
Benchmark : sigmap-v8.23-main (21 repositories, including R language)
126+
Date : 2026-07-28
127127
128128
Hit@5 : 82.2% (grep-agent baseline 44.8% — 1.59× lift)
129129
Token reduction: 96.8% (across 21 repos)

‎benchmarks/latest.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
2-
"benchmark_id": "sigmap-v8.22-main",
3-
"benchmark_date": "2026-07-27",
2+
"benchmark_id": "sigmap-v8.23-main",
3+
"benchmark_date": "2026-07-28",
44
"source": "benchmarks/reports/{benchmark-matrix,task-benchmark,token-reduction,honest-baseline}.json",
55
"repos_token": 21,
66
"repos_retrieval": 18,

‎benchmarks/reports/benchmark-matrix.json‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"generated": "2026-07-27T22:47:08.524Z",
2+
"generated": "2026-07-28T19:29:15.129Z",
33
"success": true,
44
"steps": [
55
{
@@ -11,7 +11,7 @@
1111
],
1212
"status": 0,
1313
"ok": true,
14-
"durationMs": 22933,
14+
"durationMs": 31016,
1515
"stdoutTail": "n 34s | 10.6s | **19min 23s** | 1min 57s | 1.1s | **1min 56s** |\n| **serilog** | 1min 38s | 3.6s | **1min 34s** | 9.8s | 0.4s | **9.4s** |\n| **riverpod** | 6min 14s | 10.7s | **6min 3s** | 37.4s | 1.1s | **36.3s** |\n| **okhttp** | 15.6s | 0.9s | **14.8s** | 1.6s | <0.1s | **1.5s** |\n| **laravel** | 13min 60s | 9.4s | **13min 50s** | 1min 24s | 0.9s | **1min 23s** |\n| **akka** | 6min 35s | 5.8s | **6min 30s** | 39.5s | 0.6s | **39.0s** |\n| **vapor** | 1min 26s | 1.6s | **1min 24s** | 8.6s | 0.2s | **8.4s** |\n| **vue-core** | 3min 34s | 5.3s | **3min 29s** | 21.4s | 0.5s | **20.9s** |\n| **svelte** | 3min 39s | 10.9s | **3min 28s** | 21.9s | 1.1s | **20.8s** |\n| **fastify** | 27.2s | 1.6s | **25.6s** | 2.7s | 0.2s | **2.6s** |\n| **fastapi** | 1min 29s | 1.9s | **1min 27s** | 8.9s | 0.2s | **8.7s** |\n| **ggplot2** | 3min 11s | 9.5s | **3min 1s** | 19.1s | 0.9s | **18.1s** |\n| **dplyr** | 1min 13s | 4.2s | **1min 8s** | 7.3s | 0.4s | **6.8s** |\n| **shiny** | 2min 12s | 4.4s | **2min 8s** | 13.2s | 0.4s | **12.8s** |\n\n*At 10 calls/day per repo: **1hr 50min** saved across all repos per round · **18hr 25min/day** · **6,720 hr/year***\n\nReport saved → benchmarks/reports/token-reduction.json\n",
1616
"stderrTail": ""
1717
},
@@ -23,7 +23,7 @@
2323
],
2424
"status": 0,
2525
"ok": true,
26-
"durationMs": 20361,
26+
"durationMs": 19434,
2727
"stdoutTail": " 18 28% 100% 3.6× 5/5 0/5 0/5\nlaravel 1533 271 0% 100% 306.6× 4/5 1/5 0/5\nakka 211 108 2% 100% 42.2× 3/5 2/5 0/5\nvapor 131 58 4% 0% 0× 0/5 0/5 5/5\nvue-core 232 129 2% 100% 46.4× 4/5 1/5 0/5\nsvelte 370 141 1% 80% 59.2× 3/5 1/5 1/5\nfastify 31 28 16% 63% 3.9× 5/5 0/5 3/5\nfastapi 48 25 10% 80% 7.7× 3/5 1/5 1/5\n──────────────────────────────────────────────────────────────────────────────────────────\nAVERAGE 16.0% 81.9% 5.1× 68/105 18/105 19/105\n──────────────────────────────────────────────────────────────────────────────────────────\n\nMethodology:\n Random baseline = min(1, 5/fileCount) — probability of a random 5-file selection containing the target\n Correct = target file is rank-1 result\n Partial = target file in ranks 2–5\n Wrong = target file not in top-5\n No LLM API used. All scores are retrieval rank math.\n\n[saved] /Users/manojmallick/sigmap/benchmarks/reports/retrieval.json\n",
2828
"stderrTail": "[run] express ...\n[run] flask ...\n[run] gin ...\n[run] spring-petclinic ...\n[run] rails ...\n[run] axios ...\n[run] rust-analyzer ...\n[run] abseil-cpp ...\n[run] serilog ...\n[run] riverpod ...\n[run] okhttp ...\n[run] laravel ...\n[run] akka ...\n[run] vapor ...\n[run] vue-core ...\n[run] svelte ...\n[run] fastify ...\n[run] fastapi ...\n[skip] ggplot2: no task file\n[skip] dplyr: no task file\n[skip] shiny: no task file\n"
2929
},
@@ -35,7 +35,7 @@
3535
],
3636
"status": 0,
3737
"ok": true,
38-
"durationMs": 19370,
38+
"durationMs": 23134,
3939
"stdoutTail": "738 | **$22.15** |\n| **laravel** | $41.99 | $0.471 | $41.52 | **$1245.49** |\n| **akka** | $19.76 | $0.289 | $19.47 | **$584.25** |\n| **vapor** | $4.28 | $0.078 | $4.21 | **$126.19** |\n| **vue-core** | $10.69 | $0.263 | $10.43 | **$312.78** |\n| **svelte** | $10.95 | $0.547 | $10.41 | **$312.23** |\n| **fastify** | $1.36 | $0.081 | $1.28 | **$38.39** |\n| **fastapi** | $4.46 | $0.096 | $4.36 | **$130.90** |\n| **ggplot2** | $9.54 | $0.475 | $9.06 | **$271.85** |\n| **dplyr** | $3.63 | $0.210 | $3.42 | **$102.51** |\n| **shiny** | $6.62 | $0.219 | $6.40 | **$191.88** |\n\n*Total GPT-4o savings: ~$9941.94/month across 21 repos at 10 calls/day*\n\n### 5. API cost savings by model (10 calls/day)\n\n| Model | Input $/1M | Saved / day | Saved / month |\n|------|:----------:|:-----------:|:-------------:|\n| GPT-4o | $2.50 | $331.40 | **$9941.94** |\n| Claude Sonnet | $3.00 | $397.68 | **$11930.33** |\n| Claude Haiku | $1.00 | $132.56 | **$3976.78** |\n\n*Savings scale with each model's input rate — token reduction is model-agnostic; the dollar figure is not. Verified 2026-07 pricing: GPT-4o $2.50, Claude Sonnet $3.00, Claude Haiku $1.00 per 1M input tokens.*\n\nReport saved → benchmarks/reports/quality.json\n",
4040
"stderrTail": ""
4141
},
@@ -47,7 +47,7 @@
4747
],
4848
"status": 0,
4949
"ok": true,
50-
"durationMs": 189,
50+
"durationMs": 159,
5151
"stdoutTail": " flask 26.3% 88% 3.3×\n gin 4.7% 100% 21.3×\n spring-petclinic 38.5% 100% 2.6×\n rails 0.4% 100% 250.0×\n axios 20.0% 50% 2.5×\n rust-analyzer 0.8% 100% 125.0×\n abseil-cpp 0.7% 100% 142.9×\n serilog 5.1% 20% 3.9×\n riverpod 1.1% 100% 90.9×\n okhttp 27.8% 100% 3.6×\n laravel 0.3% 100% 333.3×\n akka 2.4% 100% 41.7×\n vapor 3.8% 0% 0.0×\n vue-core 2.2% 100% 45.5×\n svelte 1.4% 80% 57.1×\n fastify 16.1% 63% 3.9×\n fastapi 10.4% 80% 7.7×\n\n Hallucination risk proxy (dark symbols)\n\n Without SigMap: 85% of codebase symbols hidden from AI\n With SigMap: 15% of indexed symbols grounded in signatures\n (10,216 grounded, 57,187 dark — across 21 repos)\n\n──────────────────────────────────────────────────────────────────────────\n\n Report saved → benchmarks/reports/task-benchmark.json\n\n",
5252
"stderrTail": "[task-benchmark] saved → /Users/manojmallick/sigmap/benchmarks/reports/task-benchmark.json\n"
5353
}

‎benchmarks/reports/honest-baseline.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"generated": "2026-07-27T22:48:32.469Z",
2+
"generated": "2026-07-28T19:29:52.966Z",
33
"methodology": {
44
"corpus": "benchmarks/tasks/*.jsonl",
55
"scorer": "src/eval/scorer.js hitAtK/reciprocalRank, k=5",

‎benchmarks/reports/quality.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
2-
"version": "8.22.0",
3-
"timestamp": "2026-07-27T22:47:08.316Z",
2+
"version": "8.23.0",
3+
"timestamp": "2026-07-28T19:29:14.955Z",
44
"assumptions": {
55
"callsPerDay": 10,
66
"models": [

‎benchmarks/reports/retrieval.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"generated": "2026-07-27T22:46:48.940Z",
2+
"generated": "2026-07-28T19:28:51.814Z",
33
"repos": [
44
{
55
"repo": "express",

‎benchmarks/reports/task-benchmark.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
{
2-
"generated": "2026-07-27T22:47:08.482Z",
2+
"generated": "2026-07-28T19:29:15.085Z",
33
"methodology": {
44
"promptWeights": {
55
"correct": 1,

‎benchmarks/reports/token-reduction.json‎

Lines changed: 23 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
2-
"version": "8.22.0",
3-
"timestamp": "2026-07-27T22:46:28.588Z",
2+
"version": "8.23.0",
3+
"timestamp": "2026-07-28T19:28:32.387Z",
44
"assumptions": {
55
"tokensPerSecUncached": 2000,
66
"cacheSpeedup": 10,
@@ -16,7 +16,7 @@
1616
"reductionPct": 98.1,
1717
"fileCount": 38,
1818
"droppedCount": 0,
19-
"timestamp": "2026-07-27T22:46:06.315Z",
19+
"timestamp": "2026-07-28T19:28:02.158Z",
2020
"timings": {
2121
"rawColdSec": 35.29,
2222
"sigColdSec": 0.66,
@@ -36,7 +36,7 @@
3636
"reductionPct": 96.1,
3737
"fileCount": 51,
3838
"droppedCount": 18,
39-
"timestamp": "2026-07-27T22:46:06.827Z",
39+
"timestamp": "2026-07-28T19:28:02.807Z",
4040
"timings": {
4141
"rawColdSec": 69.73,
4242
"sigColdSec": 2.7,
@@ -56,7 +56,7 @@
5656
"reductionPct": 95.1,
5757
"fileCount": 118,
5858
"droppedCount": 40,
59-
"timestamp": "2026-07-27T22:46:07.426Z",
59+
"timestamp": "2026-07-28T19:28:03.621Z",
6060
"timings": {
6161
"rawColdSec": 110.62,
6262
"sigColdSec": 5.45,
@@ -76,7 +76,7 @@
7676
"reductionPct": 96.8,
7777
"fileCount": 52,
7878
"droppedCount": 13,
79-
"timestamp": "2026-07-27T22:46:07.855Z",
79+
"timestamp": "2026-07-28T19:28:04.824Z",
8080
"timings": {
8181
"rawColdSec": 47.05,
8282
"sigColdSec": 1.52,
@@ -96,7 +96,7 @@
9696
"reductionPct": 98.7,
9797
"fileCount": 1193,
9898
"droppedCount": 918,
99-
"timestamp": "2026-07-27T22:46:09.362Z",
99+
"timestamp": "2026-07-28T19:28:06.490Z",
100100
"timings": {
101101
"rawColdSec": 749.53,
102102
"sigColdSec": 9.6,
@@ -116,7 +116,7 @@
116116
"reductionPct": 95.4,
117117
"fileCount": 64,
118118
"droppedCount": 14,
119-
"timestamp": "2026-07-27T22:46:10.175Z",
119+
"timestamp": "2026-07-28T19:28:08.332Z",
120120
"timings": {
121121
"rawColdSec": 36.16,
122122
"sigColdSec": 1.65,
@@ -136,7 +136,7 @@
136136
"reductionPct": 99.4,
137137
"fileCount": 675,
138138
"droppedCount": 513,
139-
"timestamp": "2026-07-27T22:46:15.043Z",
139+
"timestamp": "2026-07-28T19:28:17.424Z",
140140
"timings": {
141141
"rawColdSec": 1765.57,
142142
"sigColdSec": 10.46,
@@ -156,7 +156,7 @@
156156
"reductionPct": 99.1,
157157
"fileCount": 703,
158158
"droppedCount": 556,
159-
"timestamp": "2026-07-27T22:46:15.868Z",
159+
"timestamp": "2026-07-28T19:28:19.654Z",
160160
"timings": {
161161
"rawColdSec": 1173.76,
162162
"sigColdSec": 10.61,
@@ -176,7 +176,7 @@
176176
"reductionPct": 96.4,
177177
"fileCount": 195,
178178
"droppedCount": 117,
179-
"timestamp": "2026-07-27T22:46:16.500Z",
179+
"timestamp": "2026-07-28T19:28:21.254Z",
180180
"timings": {
181181
"rawColdSec": 97.76,
182182
"sigColdSec": 3.56,
@@ -196,7 +196,7 @@
196196
"reductionPct": 97.1,
197197
"fileCount": 465,
198198
"droppedCount": 350,
199-
"timestamp": "2026-07-27T22:46:17.530Z",
199+
"timestamp": "2026-07-28T19:28:23.017Z",
200200
"timings": {
201201
"rawColdSec": 373.61,
202202
"sigColdSec": 10.65,
@@ -216,7 +216,7 @@
216216
"reductionPct": 94.4,
217217
"fileCount": 18,
218218
"droppedCount": 0,
219-
"timestamp": "2026-07-27T22:46:18.141Z",
219+
"timestamp": "2026-07-28T19:28:23.953Z",
220220
"timings": {
221221
"rawColdSec": 15.64,
222222
"sigColdSec": 0.87,
@@ -236,7 +236,7 @@
236236
"reductionPct": 98.9,
237237
"fileCount": 1536,
238238
"droppedCount": 1265,
239-
"timestamp": "2026-07-27T22:46:19.222Z",
239+
"timestamp": "2026-07-28T19:28:25.785Z",
240240
"timings": {
241241
"rawColdSec": 839.75,
242242
"sigColdSec": 9.42,
@@ -256,7 +256,7 @@
256256
"reductionPct": 98.5,
257257
"fileCount": 211,
258258
"droppedCount": 103,
259-
"timestamp": "2026-07-27T22:46:20.524Z",
259+
"timestamp": "2026-07-28T19:28:26.651Z",
260260
"timings": {
261261
"rawColdSec": 395.27,
262262
"sigColdSec": 5.78,
@@ -276,7 +276,7 @@
276276
"reductionPct": 98.2,
277277
"fileCount": 134,
278278
"droppedCount": 76,
279-
"timestamp": "2026-07-27T22:46:21.206Z",
279+
"timestamp": "2026-07-28T19:28:27.412Z",
280280
"timings": {
281281
"rawColdSec": 85.68,
282282
"sigColdSec": 1.56,
@@ -296,7 +296,7 @@
296296
"reductionPct": 97.5,
297297
"fileCount": 249,
298298
"droppedCount": 120,
299-
"timestamp": "2026-07-27T22:46:22.232Z",
299+
"timestamp": "2026-07-28T19:28:28.148Z",
300300
"timings": {
301301
"rawColdSec": 213.77,
302302
"sigColdSec": 5.25,
@@ -316,7 +316,7 @@
316316
"reductionPct": 95,
317317
"fileCount": 370,
318318
"droppedCount": 229,
319-
"timestamp": "2026-07-27T22:46:23.459Z",
319+
"timestamp": "2026-07-28T19:28:29.396Z",
320320
"timings": {
321321
"rawColdSec": 219.09,
322322
"sigColdSec": 10.94,
@@ -336,7 +336,7 @@
336336
"reductionPct": 94.1,
337337
"fileCount": 28,
338338
"droppedCount": 0,
339-
"timestamp": "2026-07-27T22:46:24.027Z",
339+
"timestamp": "2026-07-28T19:28:29.846Z",
340340
"timings": {
341341
"rawColdSec": 27.21,
342342
"sigColdSec": 1.62,
@@ -356,7 +356,7 @@
356356
"reductionPct": 97.8,
357357
"fileCount": 32,
358358
"droppedCount": 7,
359-
"timestamp": "2026-07-27T22:46:24.644Z",
359+
"timestamp": "2026-07-28T19:28:30.534Z",
360360
"timings": {
361361
"rawColdSec": 89.19,
362362
"sigColdSec": 1.93,
@@ -376,7 +376,7 @@
376376
"reductionPct": 95,
377377
"fileCount": 189,
378378
"droppedCount": 75,
379-
"timestamp": "2026-07-27T22:46:26.038Z",
379+
"timestamp": "2026-07-28T19:28:31.229Z",
380380
"timings": {
381381
"rawColdSec": 190.73,
382382
"sigColdSec": 9.5,
@@ -396,7 +396,7 @@
396396
"reductionPct": 94.2,
397397
"fileCount": 97,
398398
"droppedCount": 38,
399-
"timestamp": "2026-07-27T22:46:26.895Z",
399+
"timestamp": "2026-07-28T19:28:31.701Z",
400400
"timings": {
401401
"rawColdSec": 72.54,
402402
"sigColdSec": 4.2,
@@ -416,7 +416,7 @@
416416
"reductionPct": 96.7,
417417
"fileCount": 93,
418418
"droppedCount": 39,
419-
"timestamp": "2026-07-27T22:46:28.490Z",
419+
"timestamp": "2026-07-28T19:28:32.343Z",
420420
"timings": {
421421
"rawColdSec": 132.3,
422422
"sigColdSec": 4.38,

0 commit comments

Comments
 (0)