Skip to content

Commit 1b8c8fe

Browse files
committed
Update readme.md
1 parent c6cc901 commit 1b8c8fe

1 file changed

Lines changed: 25 additions & 18 deletions

File tree

readme.md

Lines changed: 25 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,28 @@
1+
<div align="center">
2+
13
# AgingBench
24

3-
**A benchmark for measuring how AI agent reliability degrades over operational lifetimes.**
5+
[![Project Page](https://img.shields.io/badge/Project_Page-site-red)](https://agingbench.github.io/)
6+
[![Version](https://img.shields.io/badge/version-v0.3.0-green)](docs/CHANGELOG.md)
7+
[![AgingCard](https://img.shields.io/badge/AgingCard-v1.0.0-yellow)](#agingcard)
8+
[![CLI](https://img.shields.io/badge/CLI-agingbench-purple)](#install)
9+
10+
> **A longitudinal reliability benchmark suite that explores four aging mechanisms in memory-enabled AI agents.** <br>
11+
> 8 scenarios | 4 aging mechanisms | AgingCard v1.0.0 | Compression · Interference · Revision · Maintenance.
12+
13+
</div>
14+
15+
---
16+
17+
The reliability of any long-running system degrades over time: databases accumulate stale indices, software accrues technical debt, and human memory fades with age. Memory-enabled agents are no exception. Even with frozen weights, their system state continues to change as they accumulate context across sessions, and their ability to store, retrieve, and apply knowledge deteriorates in ways that standard snapshot evaluation cannot capture.
18+
19+
AgingBench is a longitudinal reliability benchmark suite organized around four aging mechanisms: compression, interference, revision, and maintenance, supporting various scenarios, models, memory policies and agent frameworks, ranging from a fully controlled runner to practical autonomous agents (e.g., Claude Code).
20+
21+
We are committed to actively maintaining this repository as a resource for evaluating the longitudinal reliability of AI agents, and we welcome contributions and suggestions from the community.
422

5-
AI agents with persistent memory (Claude Code, ChatGPT, enterprise assistants) are deployed for weeks or months. Over that time they don't just forget — they lose specific values while appearing correct, cite stale facts after updates, confuse similar entities across domains, and silently regress after routine maintenance events. Standard evaluations miss this: an agent can pass every behavioral test while its factual precision has dropped by 50%.
23+
## 📢 Updates
624

7-
AgingBench measures this degradation across four mechanisms (compression, interference, revision, maintenance), diagnoses *where in the agent's memory infrastructure* each failure originates, and points to *which component* to fix.
25+
* **v0.3.0** — Initial public release (2026-05-13). Eight scenarios (S1–S8) across the four aging mechanisms, AgingCard schema v1.0.0, `agingbench` + `agingbench-lite` CLIs in a single `pip install`, telemetry-mode post-hoc trace analysis. Full log will be updated in [docs/CHANGELOG.md](docs/CHANGELOG.md).
826

927
---
1028

@@ -29,15 +47,15 @@ Over time M_t loses facts → scores drop → AgingBench fits the aging curve.
2947
- **Aging curve `m(t)`** — score vs. session. Half-life = sessions until 50% of capability is lost.
3048
- **Memory is the independent variable** — same model with different policies produces different aging curves.
3149

32-
### The 8 scenarios
50+
### Latest supporting scenarios
3351

3452
| ID | Name | Tier | Sessions | What it tests |
3553
|----|------|:----:|:--------:|---------------|
3654
| **S1** | Research Literature | T1 | 8–20 | Fact survival under compression |
3755
| **S2** | Lifestyle Assistant | T1 | 8–10 | Constraint adherence + revision (forget, accumulator) |
3856
| **S3** | Knowledge Base | T1 | 8–12 | Decision fidelity under accumulation |
3957
| **S4** | Software Engineering | T1 | 8–12 | Code planning with retractions |
40-
| **S5** | Self-Planning Notebook | T1 | 8–20 blocks | Agent manages its own workspace files (via `ReactFileAdapter`) |
58+
| **S5** | Self-Planning Notebook | T1 | 8–20 | Agent manages its own workspace files |
4159
| **S6** | Naturalistic | T1 | 10–15 | Multi-domain recall with corrections |
4260
| **S7** | Research-Notes Coding Task | T2 | 10–20 | Production CLI (OpenHands / Claude Code) building a notes-app codebase |
4361
| **S8** | SWE-bench-Aging *(newly added)* | T2 | 8 | Production CLI on a curated chain of real Django GitHub issues |
@@ -148,19 +166,6 @@ For library use, set API keys as ordinary env vars (`os.environ["ANTHROPIC_API_K
148166

149167
---
150168

151-
## Integration adapters
152-
153-
AgingCards can feed downstream eval and observability platforms. Reference skeletons in [`prototype/examples/`](prototype/examples/):
154-
155-
- [`openai_evals_adapter.py`](prototype/examples/openai_evals_adapter.py)
156-
- [`langsmith_adapter.py`](prototype/examples/langsmith_adapter.py)
157-
- [`langfuse_adapter.py`](prototype/examples/langfuse_adapter.py)
158-
- [`mcp_adapter.py`](prototype/examples/mcp_adapter.py)
159-
160-
Each adapter is a ~100-line Python file that consumes an `aging_card.json` and emits the target system's preferred format. Sample cards in [`prototype/examples/sample_cards/`](prototype/examples/sample_cards/) let you test against without running the benchmark.
161-
162-
---
163-
164169
## Four aging mechanisms
165170

166171
1. **Compression** — write-before-query barrier destroys facts at compaction time
@@ -185,6 +190,8 @@ AgingBench treats memory policy as the independent variable: same model, differe
185190

186191
## Citation
187192

193+
If you find this work useful, please cite:
194+
188195
```bibtex
189196
@inproceedings{agingbench2026,
190197
title = {Long-Lived AI Agents Age Too: They Quietly Decay After Deployment},

0 commit comments

Comments
 (0)