|
10 | 10 | [](#install) |
11 | 11 | [](LICENSE) |
12 | 12 |
|
13 | | -> **A longitudinal reliability benchmark that explores four aging mechanisms in memory-enabled AI agents.** <br> |
14 | | -> 8 scenarios | 4 aging mechanisms | AgingCard v1.0.0 | Compression · Interference · Revision · Maintenance. |
| 13 | +> **A longitudinal reliability benchmark foundation for agent lifespan engineering.** <br> |
| 14 | +> 7+ scenarios | 4 aging mechanisms | AgingCard v1.0.0 | Compression · Interference · Revision · Maintenance. |
15 | 15 |
|
16 | 16 | </div> |
17 | 17 |
|
18 | 18 | ## Overview |
19 | 19 |
|
20 | | -The reliability of any long-running system degrades over time: databases accumulate stale indices, software accrues technical debt, and human memory fades with age. Memory-enabled agents are no exception. Even with frozen weights, their system state continues to change as they accumulate context across sessions, and their ability to store, retrieve, and apply knowledge deteriorates in ways that standard snapshot evaluation cannot capture. |
| 20 | +Long-lived AI agents are increasingly deployed as persistent operational systems, yet they are still evaluated like freshly initialized models. Day-one benchmarks miss a basic systems question: *how long does an agent remain reliable after deployment?* Even when model weights are frozen, an agent's effective state keeps changing as it compresses interaction history, retrieves from a growing memory store, revises facts after updates, and undergoes routine maintenance. Reliability is therefore a lifespan property of the full agent harness, not a snapshot property of the base model. |
21 | 21 |
|
22 | | -AgingBench is a longitudinal reliability benchmark suite organized around four aging mechanisms: compression, interference, revision, and maintenance, supporting various scenarios, models, memory policies and agent frameworks, ranging from a fully controlled runner to practical autonomous agents (e.g., Claude Code). |
| 22 | +**AgingBench** is a longitudinal reliability benchmark for **agent lifespan engineering (ALE)** — measuring not only whether deployed agents degrade, but what form the degradation takes and where repair should target. It organizes agent *aging* into four mechanisms (**compression**, **interference**, **revision**, **maintenance**) and uses temporal dependency graphs + paired counterfactual probes to produce stage-level diagnostic profiles (write, retrieval, utilization). Both scenario mode (controlled scenarios against your model) and telemetry mode (production-trace analysis) emit the same AgingCard schema, so the same vocabulary covers both pre-deployment evaluation and post-deployment observability. |
23 | 23 |
|
24 | | -We are committed to actively maintaining this repository as a resource for evaluating the longitudinal reliability of AI agents, and we welcome contributions and suggestions from the community. |
| 24 | +We are committed to actively maintaining this repository as a foundation for ALE research and engineering, and welcome contributions from the community. |
25 | 25 |
|
26 | 26 | ## 📢 Updates |
27 | 27 |
|
28 | | -* **v0.3.0** — Initial public release (2026-05-13). Eight scenarios (S1–S8) across the four aging mechanisms, AgingCard schema v1.0.0, `agingbench` + `agingbench-lite` CLIs in a single `pip install`, telemetry-mode post-hoc trace analysis. Full log will be updated in [docs/CHANGELOG.md](docs/CHANGELOG.md). |
| 28 | +* **v0.3.0** — Initial public release (2026-05-25). Seven core scenarios (S1–S7) across the four aging mechanisms, plus the *S8 SWE-bench-Aging* community extension. AgingCard schema v1.0.0, `agingbench` + `agingbench-lite` CLIs in a single `pip install`, behavioral-DAG telemetry-mode post-hoc trace analysis. Full log will be updated in [docs/CHANGELOG.md](docs/CHANGELOG.md). |
29 | 29 |
|
30 | 30 | --- |
31 | 31 |
|
@@ -65,7 +65,7 @@ Over time M_t loses facts → scores drop → AgingBench fits the aging curve. |
65 | 65 |
|
66 | 66 | Tier 1 = benchmark-driven loop; Tier 2 = external agent driving its own loop, wrapped via an adapter. Per-scenario READMEs at [`prototype/agingbench/scenarios/sN_*/README.md`](prototype/agingbench/scenarios/) cover data design, scoring pipeline, and example invocations. |
67 | 67 |
|
68 | | -> **Want a new scenario?** S8 was added in v0.3.0 and we're actively welcoming further scenario contributions — production agent deployments, domain-specific failure modes, anything that exercises a memory-aging axis we haven't covered yet. See [docs/CONTRIBUTING.md#how-to-add-a-new-scenario](docs/CONTRIBUTING.md#how-to-add-a-new-scenario) for the protocol (scenario manifest → generator → runner → tests). |
| 68 | +> **Want a new scenario?** S8 was added in v0.3.0 and we're actively welcoming further scenario contributions — production agent deployments, domain-specific failure modes, anything that exercises a memory-aging axis we haven't covered yet. See [docs/CONTRIBUTING.md#adding-a-scenario](docs/CONTRIBUTING.md#adding-a-scenario) for the protocol (scenario manifest → generator → runner → tests). |
69 | 69 |
|
70 | 70 | --- |
71 | 71 |
|
@@ -118,7 +118,7 @@ Each run writes to `prototype/experiments/results/<scenario>/<sut_id>/`: |
118 | 118 | # Lite — S1, S2, S7 × 3 seeds × Haiku-class. ~30 min, no Docker. |
119 | 119 | uv run --project prototype agingbench run --suite lite --seeds 3 --card |
120 | 120 |
|
121 | | -# Full — all 8 scenarios × default SUTs × 3 seeds. ~6 hr. S8 needs Docker. |
| 121 | +# Full — all scenarios (S1–S7 + S8 extension) × default SUTs × 3 seeds. ~6 hr. S8 needs Docker. |
122 | 122 | uv run --project prototype agingbench run --suite full --seeds 3 --card |
123 | 123 |
|
124 | 124 | # Pressure sweep — S1+S2+S5 at light/medium/heavy PressureConfig presets. |
@@ -197,7 +197,7 @@ If you find this work useful, please cite: |
197 | 197 |
|
198 | 198 | ```bibtex |
199 | 199 | @inproceedings{agingbench2026, |
200 | | - title = {Long-Lived AI Agents Age Too: They Quietly Decay After Deployment}, |
| 200 | + title = {Your Agents Are Aging Too: Agent Lifespan Engineering for Deployed Systems}, |
201 | 201 | author = {Zhu, Jianing and Ro, Yeonju and Robertson, John and Wang, Kevin and |
202 | 202 | Li, Junbo and Vikalo, Haris and Akella, Aditya and Wang, Zhangyang}, |
203 | 203 | booktitle = {Preprint}, |
|
0 commit comments