Skip to content

[recipes] Recall quality eval harness#354

Open
wefilmshit wants to merge 1 commit into
NateBJones-Projects:mainfrom
wefilmshit:contrib/recall-quality-eval-harness
Open

[recipes] Recall quality eval harness#354
wefilmshit wants to merge 1 commit into
NateBJones-Projects:mainfrom
wefilmshit:contrib/recall-quality-eval-harness

Conversation

@wefilmshit

Copy link
Copy Markdown

Contribution Type

  • Recipe (/recipes)
  • Schema (/schemas)
  • Dashboard (/dashboards)
  • Integration (/integrations)
  • Skill (/skills)
  • Repo improvement (docs, CI, templates)

What does this do?

Adds a small recall quality harness for local Open Brain instances. The harness captures a frozen baseline from match_thoughts, reruns the same query set after a recall-path change, and reports recall containment plus latency.

Requirements

  • Local Postgres with pgvector
  • Existing OB1-style thoughts table
  • Existing match_thoughts function
  • Python 3 and psql

Testing confirmation

Tested on a local OB1-shaped fixture database with synthetic thoughts. The fixture captured a baseline and compared the same query set against the unchanged recall path.

Checklist

  • I have read CONTRIBUTING.md
  • My contribution has a README.md with prerequisites, step-by-step instructions, and expected outcome
  • My metadata.json has all required fields
  • If my contribution depends on a skill or primitive, I declared it in metadata.json and linked it in the README
  • I tested this on my own Open Brain instance
  • No credentials, API keys, or secrets are included

@github-actions github-actions Bot added the recipe Contribution: step-by-step recipe label Jun 10, 2026
@github-actions

Copy link
Copy Markdown

Hey @wefilmshit — welcome to Open Brain Source! 👋

Thanks for submitting your first PR. The automated review will run shortly and check things like metadata, folder structure, and README completeness. If anything needs fixing, the review comment will tell you exactly what.

Once the automated checks pass, a human admin will review for quality and clarity. Expect a response within a few days.

If you have questions, check out CONTRIBUTING.md or open an issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

recipe Contribution: step-by-step recipe

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant