Add MS MARCO and StackOverflowQA search benchmark datasets - #13230
Open
Jeadie wants to merge 2 commits into
Open
Add MS MARCO and StackOverflowQA search benchmark datasets#13230Jeadie wants to merge 2 commits into
Jeadie wants to merge 2 commits into
Conversation
… datasets Extends the MTEB search benchmark suite with two new retrieval datasets, following the existing standard-layout loader pattern.
Contributor
✅ Pull with Spice PassedPassing checks:
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds MS MARCO and StackOverflowQA to the MTEB search benchmark suite.
Changes:
- Adds dataset selectors and MTEB repository mappings.
- Adds seven search configurations per dataset.
- Adds scheduled dispatch configurations and serialization tests.
Reviewed changes
Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
tools/testoperator/src/commands/search/dataset.rs |
Registers both MTEB repositories. |
tools/testoperator/src/args/search.rs |
Adds dataset CLI/serde selectors. |
tools/testoperator/src/args/dispatch.rs |
Tests dispatch serialization. |
tools/testoperator/dispatch/search/msmarco.yaml |
Dispatches MS MARCO variants. |
tools/testoperator/dispatch/search/stackoverflow_qa.yaml |
Dispatches StackOverflowQA variants. |
test/spicepods/search/mteb/msmarco/full_text_search-duckdb[file].yaml |
DuckDB full-text benchmark. |
test/spicepods/search/mteb/msmarco/full_text_search-cayenne[file].yaml |
Cayenne full-text benchmark. |
test/spicepods/search/mteb/msmarco/hf[all-minilm-l6-v2]-arrow.yaml |
Hugging Face/Arrow vector benchmark. |
test/spicepods/search/mteb/msmarco/hybrid[model2vec[potion-multilingual-128M]]-duckdb[file].yaml |
DuckDB hybrid benchmark. |
test/spicepods/search/mteb/msmarco/model2vec[potion-multilingual-128M]-duckdb[file].yaml |
DuckDB vector benchmark. |
test/spicepods/search/mteb/msmarco/model2vec[potion-multilingual-128M]-cayenne[file].yaml |
Cayenne vector benchmark. |
test/spicepods/search/mteb/msmarco/model2vec[potion-multilingual-128M]-s3_vectors.yaml |
S3 Vectors benchmark. |
test/spicepods/search/mteb/stackoverflow_qa/full_text_search-duckdb[file].yaml |
DuckDB full-text benchmark. |
test/spicepods/search/mteb/stackoverflow_qa/full_text_search-cayenne[file].yaml |
Cayenne full-text benchmark. |
test/spicepods/search/mteb/stackoverflow_qa/hf[all-minilm-l6-v2]-arrow.yaml |
Hugging Face/Arrow vector benchmark. |
test/spicepods/search/mteb/stackoverflow_qa/hybrid[model2vec[potion-multilingual-128M]]-duckdb[file].yaml |
DuckDB hybrid benchmark. |
test/spicepods/search/mteb/stackoverflow_qa/model2vec[potion-multilingual-128M]-duckdb[file].yaml |
DuckDB vector benchmark. |
test/spicepods/search/mteb/stackoverflow_qa/model2vec[potion-multilingual-128M]-cayenne[file].yaml |
Cayenne vector benchmark. |
test/spicepods/search/mteb/stackoverflow_qa/model2vec[potion-multilingual-128M]-s3_vectors.yaml |
S3 Vectors benchmark. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Jeadie
added
the
kind/task
Jeadie
enabled auto-merge
August 21, 2026 05:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mteb/msmarco, 7-shard corpus) and StackOverflowQA (mteb/stackoverflow-qa), following the existingMtebRepostandard-layout loader pattern used by the other datasets in the suite.test/spicepods/search/mteb/msmarco/andtest/spicepods/search/mteb/stackoverflow_qa/with the same 7 spicepod variants (HF+Arrow, full-text search on DuckDB/Cayenne, vector search on DuckDB/Cayenne/S3 Vectors, hybrid) used by the other datasets.tools/testoperator/dispatch/search/{msmarco,stackoverflow_qa}.yamldispatch configs, picked up automatically by the scheduledtestoperator dispatch searchworkflow.Test plan
cargo check -p testoperatorcargo clippy -p testoperator --no-depscargo fmt -p testoperator -- --checkcargo test -p testoperator(addedtest_msmarco_search_section_deserialization/test_stackoverflow_qa_search_section_deserialization, all existing dispatch tests still pass)testoperator dispatch searchCI run (validates the spicepods end-to-end by downloading the real datasets and running the search harness)