Model choice is currently an act of faith. Public benchmarks measure someone elses work, vendor numbers measure the vendors own, and both are argued about instead of used. Meanwhile every developer has a corpus of problems they have personally solved, with the solution attached.
That is a benchmark. It is private, it is exactly on-distribution for the person, and nobody can dispute it because it is theirs.
Shape
$ deja eval --build # derive cases from solved sessions
47 cases: the question, the files in play, and what the session concluded
$ deja eval --agent codex
38/47 reached the same conclusion median 6 turns
$ deja eval --agent claude
41/47 median 9 turns
The grading is the interesting part and it does not need an LLM in the easy cases: many past conclusions are checkable — a test that now passes, a specific value chosen, a file that ends up in a particular state.
Why this fits deja specifically
Deriving a case needs the problem, the attempts and the outcome as a sequence. That is #526s extraction again, pointed at evaluation instead of explanation. And it is cross-harness by construction — the whole point is comparing agents, which requires seeing all of them.
It also sidesteps the argument we cannot win: rather than disputing whose LongMemEval number is real, this hands people a way to measure on their own ground.
Hard parts, and they are hard
- Grading without an oracle. For a lot of cases "the same conclusion" is a judgement, and automating that judgement badly produces a benchmark that rewards verbosity.
- Running the agents at all. This means invoking Claude Code / Codex / opencode headlessly, with keys and cost. That is a much larger surface than anything deja does today, and it is where this could quietly become a different product.
- Contamination: the agent under test may recall the original session through deja itself and score by memory rather than by capability. That has to be turned off during evaluation, which is an amusing problem to have.
Why file it anyway
Because the cheap half is genuinely cheap and independently useful: case extraction. A list of "problems I have solved, with the answers" is worth having even if nothing ever runs against it — it is the raw material for #535, and it is the honest version of the personal-eval idea if the execution half proves too big.
Kill condition
Extract cases from my corpus and read them. If a case is not self-contained enough that I could hand it to a stranger, the extraction is not there and the rest is moot.
Model choice is currently an act of faith. Public benchmarks measure someone elses work, vendor numbers measure the vendors own, and both are argued about instead of used. Meanwhile every developer has a corpus of problems they have personally solved, with the solution attached.
That is a benchmark. It is private, it is exactly on-distribution for the person, and nobody can dispute it because it is theirs.
Shape
The grading is the interesting part and it does not need an LLM in the easy cases: many past conclusions are checkable — a test that now passes, a specific value chosen, a file that ends up in a particular state.
Why this fits deja specifically
Deriving a case needs the problem, the attempts and the outcome as a sequence. That is #526s extraction again, pointed at evaluation instead of explanation. And it is cross-harness by construction — the whole point is comparing agents, which requires seeing all of them.
It also sidesteps the argument we cannot win: rather than disputing whose LongMemEval number is real, this hands people a way to measure on their own ground.
Hard parts, and they are hard
Why file it anyway
Because the cheap half is genuinely cheap and independently useful: case extraction. A list of "problems I have solved, with the answers" is worth having even if nothing ever runs against it — it is the raw material for #535, and it is the honest version of the personal-eval idea if the execution half proves too big.
Kill condition
Extract cases from my corpus and read them. If a case is not self-contained enough that I could hand it to a stranger, the extraction is not there and the rest is moot.