Skip to content

Run the join test: the graph did not meet the registered bar - #18

Merged
Kcodess2807 merged 1 commit into
mainfrom
join-run
Nov 19, 2026
Merged

Kcodess2807 merged 1 commit into
mainfrom
join-run

Conversation

@Kcodess2807

Copy link
Copy Markdown
Owner

Runs the join test pre-registered in #17. The graph did not meet the registered bar.

Setup Score
Full context 28/29
Graph memory 19/29
Episodic 12/29
None 5/29
  • Registered comparison: on the 24 join tasks, graph memory alone was right on 12 and episodic on 5. Sign test p=0.143, against a bar of p<0.05.
  • Where the graph wins: questions that start from GitHub and end at a told fact. 11 of 12, where episodic got 0.
  • Where it loses: told fact first, then GitHub. 1 of 6 against 4, because the two-hop walk stops before the PR's author.
  • Grader note: four graph misses answered "RUN-7 board" for gold "RUN-7". Counting them would give p=0.013, but that reading came after seeing the answers, so the README reports it without claiming it.

Code: run_memory.py gains a full-context arm (--corpus) and prints the paired sign test. The sign test was checked against a known case (7 to 0 gives p=0.016).

🤖 Generated with Claude Code

Graph memory 19/29, episodic 12/29, full context 28/29, none 5/29. On the 24
join tasks graph memory alone was right on 12 and episodic on 5, p=0.143, so
the pre-registered p<0.05 is not met.

The graph wins every question that starts from GitHub and ends at a told fact
(11 of 12, episodic 0) and loses the reverse direction (1 of 6 against 4),
where the two-hop walk stops before the PR's author.

run_memory.py gains a full-context arm (--corpus) and prints the sign test.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
trace-rag Ready Ready Preview Nov 19, 2026 6:46am UTC

@Kcodess2807
Kcodess2807 merged commit 80c36a6 into main Nov 19, 2026
8 checks passed

This branch was successfully deployed

1 active deployment
Preview — 8483fd08 Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant