Skip to content

Commit c3fbdaa

Browse files
Siddhant-K-codeona-agent
Siddhant-K-code
and
ona-agent
authored
docs(agents-md): add AGENTS.md integration guide and repo AGENTS.md (#109)
Adds docs/agents-md-integration.md covering: what AGENTS.md is, how agent-strace reads it, change detection with agent-strace drift, baseline anchoring after AGENTS.md edits, and a CI workflow example. Also adds AGENTS.md to the repo itself so AI agents working on this codebase understand the zero-dependency constraint, storage format rules, and development workflow. Closes #96 Co-authored-by: Ona <[email protected]>
1 parent f7fb67e commit c3fbdaa

4 files changed

Lines changed: 220 additions & 1 deletion

File tree

‎AGENTS.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# AGENTS.md — agent-strace development guide
2+
3+
This file tells AI coding agents how to work with the agent-strace repository.
4+
5+
## Project layout
6+
7+
```
8+
src/agent_trace/ Core library — one module per feature
9+
tests/ One test file per module (test_<module>.py)
10+
ADRs/ Architecture Decision Records — read before adding dependencies
11+
docs/ Integration guides
12+
examples/ Usage examples for each integration
13+
pyproject.toml Package config and optional extras
14+
```
15+
16+
## Constraints
17+
18+
- **Zero runtime dependencies in `src/agent_trace/`.** Use Python stdlib only. If a feature requires a third-party package, it must be an optional extra in `pyproject.toml` under `[project.optional-dependencies]`, imported lazily inside the function with a clear `ImportError` message. See ADR-0003.
19+
- **Never change the `.agent-traces/` storage format** in a way that breaks existing sessions. Adding new fields is fine. Removing or renaming existing fields requires a major version bump and human approval.
20+
- **Every new CLI command must be registered in `cli.py`** following the existing subparser pattern.
21+
- **Every new feature must have tests** in `tests/test_<module>.py`. Run `python -m pytest tests/ -v` to verify.
22+
- **OTLP export uses HTTP/JSON only** — no gRPC. See ADR-0006.
23+
24+
## Development workflow
25+
26+
```bash
27+
# Install in editable mode
28+
pip install -e .
29+
30+
# Run tests
31+
python -m pytest tests/ -v
32+
33+
# Run a single test file
34+
python -m pytest tests/test_watch.py -v
35+
```
36+
37+
## Adding a new feature
38+
39+
1. Create `src/agent_trace/<feature>.py`
40+
2. Add a `cmd_<feature>` function following the pattern in existing modules
41+
3. Import and register it in `cli.py`
42+
4. Add new `EventType` values to `models.py` if needed
43+
5. Write tests in `tests/test_<feature>.py`
44+
6. Update README.md with the new command and an example
45+
46+
## Version bumping
47+
48+
- New feature (new CLI command, new integration, new flag): bump minor (`0.38.1` → `0.39.0`)
49+
- Bug fix or small improvement: bump patch (`0.38.1` → `0.38.2`)
50+
- Breaking change to CLI or storage format: bump major — check with maintainer first
51+
52+
Version is in `src/agent_trace/__init__.py`.
53+
54+
## ADRs to read before making architectural decisions
55+
56+
- ADR-0003: Zero runtime dependencies
57+
- ADR-0006: OTLP HTTP/JSON only (no gRPC)
58+
- ADR-0001: Flat event stream data model
59+
- ADR-0002: NDJSON file storage (no database)

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1523,6 +1523,7 @@ uv tool install -e .
15231523

15241524
## Related
15251525

1526+
- [AGENTS.md integration guide](docs/agents-md-integration.md) - how to use agent-strace with AGENTS.md for drift detection and CI gating
15261527
- [Architecture Decision Records](ADRs/) - design decisions and their rationale
15271528
- [The agent observability gap (blog)](https://siddhantkhare.com/writing/agent-observability-gap) - the problem this tool addresses
15281529
- [The agent observability gap (thread)](https://x.com/Siddhant_K_code/status/2032834557628788940) - discussion on X

‎docs/agents-md-integration.md‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
# Using agent-strace with AGENTS.md
2+
3+
## What is AGENTS.md?
4+
5+
`AGENTS.md` is a Markdown file placed at the root of a repository that tells AI coding agents how to work with the project. It specifies tool permissions, constraints, coding conventions, and model hints. Claude Code, Cursor, and other agents read it automatically at session start.
6+
7+
See the [AGENTS.md specification](https://docs.anthropic.com/en/docs/claude-code/memory) for the full format.
8+
9+
---
10+
11+
## How agent-strace reads AGENTS.md
12+
13+
agent-strace uses `AGENTS.md` as a **behavioral baseline anchor**. When a session starts, it records the current `AGENTS.md` hash alongside the session metadata. This lets it detect when agent behavior changes after an `AGENTS.md` edit — distinguishing intentional config changes from unexpected drift.
14+
15+
Specifically, agent-strace reads:
16+
17+
| Field | How it's used |
18+
|---|---|
19+
| Tool permissions (`allowed_tools`, `disallowed_tools`) | Checked against actual tool calls in `agent-strace audit` |
20+
| File path constraints (`read_only_paths`, `write_paths`) | Checked in `agent-strace audit --policy` |
21+
| Model hints (`model`) | Compared against `LLM_REQUEST` events in drift detection |
22+
| Any free-text constraints | Indexed for `agent-strace optimize` suggestions |
23+
24+
---
25+
26+
## Change detection
27+
28+
`agent-strace drift` computes a behavioral fingerprint for each session (tool mix, error rate, retry rate, blast radius) and compares it against a saved baseline. When `AGENTS.md` changes, the baseline should be updated so drift is measured from the new intended behavior, not the old one.
29+
30+
```bash
31+
# Check for behavioral drift since the last baseline
32+
agent-strace drift
33+
34+
# Check drift specifically after an AGENTS.md edit
35+
agent-strace drift --baseline .agent-strace/baseline.json
36+
```
37+
38+
The drift report shows which dimensions changed and by how much:
39+
40+
```
41+
Behavioral drift report
42+
=======================
43+
Baseline: .agent-strace/baseline.json (saved 2026-05-01)
44+
Sessions compared: 12
45+
46+
error_rate 0.04 → 0.12 ▲ +200% ⚠ DRIFT
47+
retry_rate 0.08 → 0.09 ▲ +12% OK
48+
blast_radius 3.2 → 3.1 ▼ -3% OK
49+
tool_mix 0.91 similarity OK
50+
51+
Verdict: DRIFT DETECTED — error_rate increased significantly.
52+
Run `agent-strace explain` on recent sessions to investigate.
53+
```
54+
55+
---
56+
57+
## Baseline anchoring
58+
59+
After updating `AGENTS.md`, save a new baseline so future drift is measured from the new intended behavior:
60+
61+
```bash
62+
# Save a baseline after an AGENTS.md update
63+
agent-strace drift --save-baseline .agent-strace/baseline.json
64+
65+
# Tag it for traceability
66+
git add .agent-strace/baseline.json
67+
git commit -m "chore: update behavioral baseline after AGENTS.md change"
68+
```
69+
70+
The baseline file is a small JSON document (< 1 KB) that records the statistical distribution of behavioral metrics across recent sessions. It is safe to commit.
71+
72+
---
73+
74+
## CI integration
75+
76+
Gate on behavioral regression whenever `AGENTS.md` changes. Add this to your CI workflow:
77+
78+
```yaml
79+
# .github/workflows/agent-behavior.yml
80+
name: Agent behavior gate
81+
82+
on:
83+
push:
84+
paths:
85+
- AGENTS.md
86+
- .agent-strace/baseline.json
87+
88+
jobs:
89+
drift-check:
90+
runs-on: ubuntu-latest
91+
steps:
92+
- uses: actions/checkout@v4
93+
94+
- name: Install agent-strace
95+
run: pip install agent-strace
96+
97+
- name: Check behavioral drift
98+
run: |
99+
agent-strace drift \
100+
--baseline .agent-strace/baseline.json \
101+
--save-baseline /tmp/new-baseline.json
102+
# Fails with exit code 1 if drift exceeds threshold
103+
104+
- name: Run eval CI gate
105+
run: |
106+
agent-strace eval ci \
107+
--baseline .agent-strace/eval-baseline.json \
108+
--tolerance 1
109+
```
110+
111+
The `drift` command exits non-zero when any dimension exceeds its threshold, making it suitable as a CI gate.
112+
113+
---
114+
115+
## Recommended AGENTS.md snippet
116+
117+
Add this to your project's `AGENTS.md` to tell agents how to work with agent-strace:
118+
119+
```markdown
120+
## Observability
121+
122+
This project uses [agent-strace](https://github.com/Siddhant-K-code/agent-trace) to
123+
capture and analyse agent sessions.
124+
125+
- Traces are stored in `.agent-traces/` (gitignored)
126+
- Run `agent-strace replay` after a session to review what happened
127+
- Run `agent-strace drift` to check for behavioral regression
128+
- Run `agent-strace retention clean --max-age-days 30` periodically to free disk space
129+
130+
Do not delete `.agent-traces/` manually — use `agent-strace retention clean`.
131+
```
132+
133+
---
134+
135+
## Detecting AGENTS.md drift
136+
137+
`agent-strace drift` also detects when the agent's behavior diverges from what `AGENTS.md` specifies — for example, if the agent starts writing to paths that `AGENTS.md` marks as read-only, or calling tools that are listed as disallowed.
138+
139+
```bash
140+
# Audit the latest session against the current AGENTS.md policy
141+
agent-strace audit --policy AGENTS.md
142+
143+
# Check all sessions from the last 7 days
144+
agent-strace audit --policy AGENTS.md --since 7d
145+
```
146+
147+
---
148+
149+
## Full workflow
150+
151+
```
152+
1. Edit AGENTS.md
153+
2. Run a few test sessions with the agent
154+
3. agent-strace drift --save-baseline .agent-strace/baseline.json
155+
4. git commit -m "chore: update baseline after AGENTS.md change"
156+
5. CI gates on future drift automatically
157+
```
158+
159+
This closes the loop between "I updated the agent's instructions" and "I know the agent is actually following them."

‎src/agent_trace/__init__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
"""agent-trace: strace for AI agents."""
22

3-
__version__ = "0.42.0"
3+
__version__ = "0.42.1"

0 commit comments

Comments
 (0)