This file summarizes the headline measurements and the interpretation limits
for the current MS MARCO GenQA experiment line. Detailed stage notes are kept
in docs/experiments.md.
Replacing BM25 top-3 passages with cross-encoder-reranked dense top-3 passages improves generation metrics under a paired setup:
| Retrieval source -> T5-small | ROUGE-L | BLEU | EM | Token-F1 |
|---|---|---|---|---|
| BM25 | 0.1859 | 0.0717 | 0.0135 | 0.1966 |
| Reranked | 0.3621 | 0.2922 | 0.0606 | 0.3677 |
| Delta | +0.1763 | +0.2206 | +0.0471 | +0.1711 |
The comparison uses the same generator, prompt format, top-k depth, and paired query set.
Paired-bootstrap confidence intervals on 6,980 paired queries:
| Metric | Delta | 95% CI |
|---|---|---|
| ROUGE-L | +0.1742 | [+0.1663, +0.1820] |
| BLEU | +0.1330 | [+0.1265, +0.1395] |
| Exact Match | +0.0471 | [+0.0417, +0.0527] |
| Token-F1 | +0.1711 | [+0.1632, +0.1789] |
All intervals are above zero.
Measured on dev/small with the full passage corpus:
| Metric | Value |
|---|---|
| MRR@10 | 0.1703 |
| Recall@100 | 0.6212 |
| Recall@1000 | 0.8154 |
The dense stage is evaluated on a controlled 50k-passage sample, not the full 8.8M-passage corpus.
| Metric | BM25-on-sample | Dense |
|---|---|---|
| MRR@10 | 0.6948 | 0.8830 |
| Recall@100 | 0.9338 | 0.9946 |
Reranking reorders dense top-100 results:
| Metric | Dense | Reranked | Delta |
|---|---|---|---|
| MRR@10 | 0.8830 | 0.9304 | +0.0474 |
| nDCG@10 | 0.9041 | 0.9434 | +0.0393 |
| Recall@100 | 0.9946 | 0.9946 | +0.0000 |
Recall@100 is unchanged because reranking only changes order within the retrieved top-100.
The full-corpus BM25 first stage and fixed top-100 cross-encoder reranker were also evaluated on the deeply judged TREC-DL passage tracks:
| Track | Metric | BM25 | BM25 + CE | Delta |
|---|---|---|---|---|
| 2019 (43 topics) | MRR@10, rel >= 2 | 0.5471 | 0.8787 | +0.3315 |
| 2019 (43 topics) | nDCG@10, graded | 0.4239 | 0.7210 | +0.2971 |
| 2020 (54 topics) | MRR@10, rel >= 2 | 0.6280 | 0.8256 | +0.1976 |
| 2020 (54 topics) | nDCG@10, graded | 0.4773 | 0.6801 | +0.2027 |
All judged topics remain in the denominator, and an independent ir-measures
cross-check reproduced the metrics with a maximum absolute delta of
2.22e-16. The complete protocol, runtime notes, query-level lift analysis,
and artifact hashes are in
docs/trec_dl_external_validity.md.
These are validated retrieval results. They do not show that the retrieval gain transfers to answer generation on TREC-DL.
The same BM25 first stage and unchanged
cross-encoder/ms-marco-MiniLM-L-6-v2 reranker were evaluated on two BEIR test
collections with their own corpora and qrels. The reranker reorders the fixed
BM25 top-100 candidate set:
| Dataset | System | MRR@10 | nDCG@10 | Recall@100 | Recall@1000 |
|---|---|---|---|---|---|
| NFCorpus (323 queries) | BM25 | 0.5186 | 0.3064 | 0.2378 | 0.4572 |
| NFCorpus (323 queries) | BM25 + CE | 0.5662 | 0.3411 | 0.2378 | n/a (top-100 run) |
| SciFact (300 queries) | BM25 | 0.6312 | 0.6617 | 0.8759 | 0.9606 |
| SciFact (300 queries) | BM25 + CE | 0.6517 | 0.6787 | 0.8759 | n/a (top-100 run) |
Cross-encoder reranking raises MRR@10 and nDCG@10 on both datasets. The larger relative lift is on NFCorpus: +9.18% MRR@10 and +11.33% nDCG@10, compared with +3.25% and +2.56% on SciFact. Recall@100 is unchanged by construction because the candidate membership is fixed. The first-stage limitation is much stronger on NFCorpus (Recall@100 0.2378) than on SciFact (0.8759), so reranking alone cannot recover most NFCorpus relevant documents. The follow-up below shows that this is a benchmark-level candidate limitation; it should not be attributed to retriever capacity alone.
All 323 NFCorpus and 300 SciFact judged queries are included. Independent
ir-measures evaluation reproduced every reported metric to floating-point
precision (maximum absolute delta 4.45e-16), and the run-file audit found no
missing topics, malformed rows, duplicate documents, or candidate-set changes.
The protocol, runtime evidence, provenance, and metric boundaries are recorded
in docs/cross_domain_benchmarks.md.
The exact four run files are recoverable without private credentials from the
checksummed
v2.2-beir-cross-domain-baselines
release. make reproduce-beir-eval verifies the archive and member hashes,
recovers public qrels through ir_datasets, and recomputes all reported rows
without rebuilding indexes or rerunning the cross-encoder.
These results show that the ranking benefit transfers to two non-MS-MARCO retrieval collections. They do not establish broad cross-domain RAG generalization or downstream generation quality.
Query-level analysis of the fixed NFCorpus BM25 run separates candidate-set absence from partial coverage:
| Diagnostic | Queries | Share |
|---|---|---|
| At least one relevant document in BM25 top 100 | 251 | 77.7% |
| No relevant document in BM25 top 100 | 72 | 22.3% |
| First relevant hit only at ranks 101-1000 | 24 | 7.4% |
| No relevant hit at depth 1000 | 48 | 14.9% |
| Complete relevant-document coverage at 100 | 19 | 5.9% |
Extending the same BM25 output from depth 100 to 1000 increases macro Recall
from 0.2378 to 0.4572 and finds 3,616 additional positive qrels across 200
queries. It still leaves 6,753 of 12,334 positive qrels unretrieved. The
definitions, exact reconciliation, limitations, and reproduction command are
in
docs/nfcorpus_first_stage_error_analysis.md.
A complete review of the 24 depth-recoverable and 48 top-1000-miss queries
assigned 67/72 cases to source_context_dependency, 4/72 to
vocabulary_or_form_mismatch, and 1/72 to
underspecified_or_ambiguous_query. Sixty-two cases contain only relevance
level 1 qrels, and topic pages account for 58/72 no-hit-at-100 queries. In the
fixed source-type slices, the unchanged cross-encoder improves early ranking
for every group; on the 144 non-topic queries MRR@10 moves from 0.5073 to
0.5827 and nDCG@10 from 0.2612 to 0.3224 with Recall@100 fixed at 0.2769.
The defensible conclusion is therefore narrower than "BM25 fails on medical
language": many NFCorpus failures depend on link-derived relevance that is not
recoverable from the exported page title alone. The case-level evidence,
taxonomy contract, limitations, and exact reproduction command are in
docs/nfcorpus_first_stage_taxonomy_review.md.
The matching SciFact analysis shows a different failure shape:
| Diagnostic | Queries | Share |
|---|---|---|
| At least one relevant document in BM25 top 100 | 265 | 88.3% |
| No relevant document in BM25 top 100 | 35 | 11.7% |
| First relevant hit only at ranks 101-1000 | 24 | 8.0% |
| No relevant hit at depth 1000 | 11 | 3.7% |
| Complete relevant-document coverage at 100 | 259 | 86.3% |
Extending the same BM25 output from depth 100 to 1000 increases macro Recall
from 0.8759 to 0.9606 and finds 28 additional positive qrels across 28
queries. It leaves 13 of 339 positive qrels unretrieved. Unlike NFCorpus,
most SciFact queries already have complete relevant-document coverage inside
the fixed top-100 candidate set, so the large NFCorpus candidate-set ceiling
does not repeat at the same scale. The definitions, exact reconciliation,
limitations, and reproduction command are in
docs/scifact_first_stage_error_analysis.md.
The bounded review of the 35 SciFact queries with no judged relevant document in BM25 top 100 shows a narrower failure mode than NFCorpus:
| Primary label | Queries | Share |
|---|---|---|
terminology_or_evidence_form_mismatch |
28 | 80.0% |
lexical_competition_at_depth_cutoff |
4 | 11.4% |
short_or_broad_claim |
3 | 8.6% |
All 35 residual cases show top_lexical_competition: the strongest BM25
candidates match more claim tokens than the judged positive evidence. In
31/35 cases, the judged positive evidence covers at most 25% of the claim
content tokens; 13/35 contain negation, direction, activation, inhibition, or
comparative cues; and 10 queries share one of five judged-positive evidence
documents with another residual claim.
This review supports keeping the pipeline frozen for the report. The remaining
SciFact misses point to scientific claim/evidence formulation and candidate
depth limits, not to a reranker or generator change. The case-level output,
drift contract, limitations, and reproduction command are in
docs/scifact_failure_review.md.
The combined NFCorpus/SciFact error analysis separates candidate-set absence, ranking-depth misses, and query/dataset effects under the same fixed BM25 top-100 reranker condition:
| Partition | NFCorpus | SciFact | Interpretation |
|---|---|---|---|
| No relevant document in top 100 | 72 (22.3%) | 35 (11.7%) | The reranker is more often given no relevant candidate on NFCorpus. |
| First relevant hit only at ranks 101-1000 | 24 (7.4%) | 24 (8.0%) | Both datasets have a comparable depth-recoverable query count. |
| No relevant hit at depth 1000 | 48 (14.9%) | 11 (3.7%) | The residual lexical first-stage tail is larger on NFCorpus. |
| Partial top-100 coverage | 232 (71.8%) | 6 (2.0%) | NFCorpus loses much more positive-qrel mass outside the candidate set. |
| Complete top-100 coverage | 19 (5.9%) | 259 (86.3%) | SciFact usually gives the reranker a complete judged candidate set. |
SciFact exceeds NFCorpus by +0.6381 Recall@100 and by +80.5 percentage
points in complete top-100 coverage share. The complete NFCorpus manual review
labels 67/72 no-hit-at-100 cases as source_context_dependency; the SciFact
residual review instead labels 28/35 misses as
terminology_or_evidence_form_mismatch. The evidence therefore supports
keeping the pipeline frozen while separating dataset/query-form effects from
retrieval-capacity changes. The machine-checked contract, reproduction command,
and limitations are in
docs/cross_dataset_error_analysis.md.
The predeclared follow-up holds the 102-query test/video cohort, corpus, qrels, BM25 index, retriever parameters, reranker, and architecture fixed while changing only the query text:
| Representation | System | MRR@10 | nDCG@10 | Recall@100 | Recall@1000 |
|---|---|---|---|---|---|
| Title | BM25 | 0.4780 | 0.2493 | 0.2821 | 0.4919 |
| Description | BM25 | 0.5308 | 0.2929 | 0.3272 | 0.6413 |
| Title + description | BM25 | 0.6036 | 0.3457 | 0.3700 | 0.6723 |
| Title | BM25 + CE | 0.5220 | 0.2980 | 0.2821 | n/a |
| Description | BM25 + CE | 0.6357 | 0.3568 | 0.3272 | n/a |
| Title + description | BM25 + CE | 0.6689 | 0.3853 | 0.3700 | n/a |
For the primary Recall@100 comparison, description alone improves the point
estimate by +0.0451, but its paired 95% interval crosses zero
([-0.0024, +0.0943], p = 0.0610). Title plus description gives a
+0.0880 improvement with interval [+0.0535, +0.1265] and
p < 0.0002. It reduces no-hit-at-100 queries from 11 to 4 and no-hit-at-1000
queries from 8 to 0.
All 306 reranker candidate-set checks pass, and independent ir-measures
evaluation agrees with the project evaluator within 1e-12. The result
supports a source-context limitation on this official video subset, not a
general architecture, cross-dataset, generation, or grounding claim. The
complete paired results and interpretation boundary are in
docs/reports/2026-07-28-nfcorpus-video-query-representation.md.
Token-F1 lift by query type:
| Query type | n | BM25 | Reranked | Delta |
|---|---|---|---|---|
| DESCRIPTION | 3,725 | 0.1889 | 0.3939 | +0.2050 |
| ENTITY | 631 | 0.1765 | 0.3186 | +0.1421 |
| LOCATION | 498 | 0.2495 | 0.3928 | +0.1433 |
| NUMERIC | 1,665 | 0.1997 | 0.3235 | +0.1238 |
| PERSON | 461 | 0.2186 | 0.3557 | +0.1371 |
DESCRIPTION queries benefit most; NUMERIC queries benefit least.
The main generation comparison is paired and stable, but the retrieval stages have different evaluation boundaries:
| Comparison | Interpretation |
|---|---|
| Full-corpus BM25 | Realistic lexical baseline over the full corpus |
| Dense vs BM25-on-sample | Controlled comparison on the same qrels-anchored pool |
| Dense vs reranked | Valid ordering comparison over dense top-100 |
| Dense-sample vs full-corpus BM25 | Not a direct apples-to-apples comparison |
The dense sample includes all dev relevant documents by construction. This is useful for isolating model behavior, but it is optimistic relative to full corpus retrieval.
| Status | Boundary |
|---|---|
| Validated | The paired T5-small generation comparison on MS MARCO dev/small; full-corpus BM25 plus cross-encoder retrieval on TREC-DL 2019/2020; full-corpus BM25 plus fixed top-100 cross-encoder reranking on BEIR NFCorpus and SciFact; the NFCorpus/SciFact first-stage diagnostics and bounded SciFact residual review; the bounded query-representation comparison on the official 102-query NFCorpus test/video subset. |
| Implemented but not yet evaluated | The T5-base generator-capacity sweep and configurable alternative-generator paths. Their existence is not an empirical result. |
| Not supported by current evidence | Retrieval lift transfers to generation on TREC-DL or BEIR; a fair full-corpus dense-vs-BM25 conclusion; broad cross-domain RAG generalization beyond the two evaluated retrieval collections. |
The surface metrics show a clear lift from reranked retrieval. The grounding audit should be read separately: stronger retrieval improves answer overlap, but it does not by itself prove deeper reasoning or fully faithful generation.
The project therefore reports both answer-surface metrics and
grounding-oriented checks. The detailed analysis is in docs/experiments.md
and the paper-style report.