Skip to content

Walk three hops; joins tested on a fresh repo - #19

Merged
Kcodess2807 merged 3 commits into
mainfrom
hop-limit
Nov 19, 2026
Merged

Kcodess2807 merged 3 commits into
mainfrom
hop-limit

Conversation

@Kcodess2807

Copy link
Copy Markdown
Owner

Fixes the direction the graph lost in the join test (#18), and tests the fix on a repo nothing was tuned on.

The fix

  • The walk goes three hops, and the model gets up to 60 facts.
  • A told fact is a hop of its own, so team -> file -> PR -> author needs three hops.
  • A probe with no model calls found a second cause: the cap of 20 cut the fact that answered. Either change alone left the answer missing on 4 of 6 tasks. Both together reached 6 of 6.

The test

  • 28 join tasks on a frozen snapshot of pydantic/pydantic.
  • Committed in a9573ac, before the code change and before any run.
Setup Score
Full context 22/28 (28/28 with "Lisbon" accepted for "Lisbon office")
Graph memory 27/28
Episodic 10/28
None 5/28
  • Both registered readings are met. Graph memory alone was right on 17 join tasks and episodic on none (p<0.0001). It answered 6 of 6 in the direction it lost on pytest.
  • Graph memory ties full context, it doesn't beat it. Full context's six "misses" are the grader, and the eval README says so.
  • The pytest rerun is only a check, because the fix came from that set: 19 to 23 of 29.

Not rerun: the 175 retrieval tasks. Three hops and 60 facts mean more tokens per question there, and that cost is unmeasured.

Verified: 92 engine tests pass.

🤖 Generated with Claude Code

Kcodess2807 and others added 3 commits November 19, 2026 12:28
28 join tasks on a frozen snapshot of pydantic/pydantic, which no earlier run
or fix has seen. Committed before the three-hop change and before any run.

Two readings are fixed in the task file: graph memory has to beat episodic
recall on the join tasks (sign test, p<0.05), and with three hops it should
answer at least 4 of the 6 told-then-GitHub tasks, the direction it lost on
pytest.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
A told fact is a hop of its own, so team -> file -> PR -> author needs three.
At two hops the walk reached the PR and not its author, and at a cap of 20 the
fact that answered was cut by facts from unrelated seeds. On the pytest join
tasks the answer was in the facts for 2 of 6 before and 6 of 6 after; either
change alone left it at 2.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
On 28 join tasks for a repo no run or fix had seen, written before the hop
change, both registered readings are met: graph memory alone right on 17 join
tasks and episodic on none, and 6 of 6 in the direction it lost on pytest.

Full context did as well: its six reporter answers were right but dropped the
word "office" from the gold, so the strict grader failed them. Reported as a
tie, not a win. The 175 retrieval tasks are not rerun with three hops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
trace-rag Ready Ready Preview Nov 19, 2026 7:13am UTC

@Kcodess2807
Kcodess2807 merged commit fb3cc99 into main Nov 19, 2026
8 checks passed

This branch was successfully deployed

1 active deployment
Preview — 446719d0 Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant