Skip to content

Rerun the 175 retrieval tasks with three hops - #20

Merged
Kcodess2807 merged 1 commit into
mainfrom
retrieval-3hop
Nov 19, 2026
Merged

Kcodess2807 merged 1 commit into
mainfrom
retrieval-3hop

Conversation

@Kcodess2807

Copy link
Copy Markdown
Owner

Reruns the 175 retrieval tasks with the three-hop walk from #19, to check it cost nothing.

click before click after pytest before pytest after
Full context 80/81 80/81 91/94 90/94
BM25 75/81 75/81 80/94 82/94
Graph 69/81 69/81 85/94 84/94
Vector 42/81 41/81 46/94 47/94
Graph prompt tokens 2,379 2,849 10,637 10,433
  • No accuracy change beyond noise. Click got the same 69 tasks. Pytest gained one and lost two.
  • Graph and BM25 are still tied.
  • Cost: about 20% more prompt tokens per graph question on click, and none on pytest.

Adds the result files and a table in the eval README.

🤖 Generated with Claude Code

Nothing moved beyond noise. Graph 69/81 on click (unchanged) and 84/94 on
pytest (85 before, one gained and two lost). Graph and BM25 stay tied, 153 to
157. Prompt tokens per graph question rose about 20% on click and not on pytest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
trace-rag Ready Ready Preview Nov 19, 2026 7:31am UTC

@Kcodess2807
Kcodess2807 merged commit b40b9cc into main Nov 19, 2026
8 checks passed

This branch was successfully deployed

1 active deployment
Preview — a18965f7 Deployed Nov 19, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant