-
Notifications
You must be signed in to change notification settings - Fork 0
Expand file tree
/
Copy pathreport.html
More file actions
4042 lines (4040 loc) · 190 KB
/
Copy pathreport.html
File metadata and controls
4042 lines (4040 loc) · 190 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" lang="" xml:lang="">
<head>
<meta charset="utf-8" />
<meta name="generator" content="pandoc" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
<title>MS MARCO Generative QA Repository Report</title>
<style>
/* Default styles provided by pandoc.
** See https://pandoc.org/MANUAL.html#variables-for-html for config info.
*/
html {
color: #1a1a1a;
background-color: #fdfdfd;
}
body {
margin: 0 auto;
max-width: 36em;
padding-left: 50px;
padding-right: 50px;
padding-top: 50px;
padding-bottom: 50px;
hyphens: auto;
overflow-wrap: break-word;
text-rendering: optimizeLegibility;
font-kerning: normal;
}
@media (max-width: 600px) {
body {
font-size: 0.9em;
padding: 12px;
}
h1 {
font-size: 1.8em;
}
}
@media print {
html {
background-color: white;
}
body {
background-color: transparent;
color: black;
font-size: 12pt;
}
p, h2, h3 {
orphans: 3;
widows: 3;
}
h2, h3, h4 {
page-break-after: avoid;
}
}
p {
margin: 1em 0;
}
a {
color: #1a1a1a;
}
a:visited {
color: #1a1a1a;
}
img {
max-width: 100%;
}
svg {
height: auto;
max-width: 100%;
}
h1, h2, h3, h4, h5, h6 {
margin-top: 1.4em;
}
h5, h6 {
font-size: 1em;
font-style: italic;
}
h6 {
font-weight: normal;
}
ol, ul {
padding-left: 1.7em;
margin-top: 1em;
}
li > ol, li > ul {
margin-top: 0;
}
blockquote {
margin: 1em 0 1em 1.7em;
padding-left: 1em;
border-left: 2px solid #e6e6e6;
color: #606060;
}
code {
font-family: Menlo, Monaco, Consolas, 'Lucida Console', monospace;
font-size: 85%;
margin: 0;
hyphens: manual;
}
pre {
margin: 1em 0;
overflow: auto;
}
pre code {
padding: 0;
overflow: visible;
overflow-wrap: normal;
}
.sourceCode {
background-color: transparent;
overflow: visible;
}
hr {
border: none;
border-top: 1px solid #1a1a1a;
height: 1px;
margin: 1em 0;
}
table {
margin: 1em 0;
border-collapse: collapse;
width: 100%;
overflow-x: auto;
display: block;
font-variant-numeric: lining-nums tabular-nums;
}
table caption {
margin-bottom: 0.75em;
}
tbody {
margin-top: 0.5em;
border-top: 1px solid #1a1a1a;
border-bottom: 1px solid #1a1a1a;
}
th {
border-top: 1px solid #1a1a1a;
padding: 0.25em 0.5em 0.25em 0.5em;
}
td {
padding: 0.125em 0.5em 0.25em 0.5em;
}
header {
margin-bottom: 4em;
text-align: center;
}
#TOC li {
list-style: none;
}
#TOC ul {
padding-left: 1.3em;
}
#TOC > ul {
padding-left: 0;
}
#TOC a:not(:hover) {
text-decoration: none;
}
code{white-space: pre-wrap;}
span.smallcaps{font-variant: small-caps;}
div.columns{display: flex; gap: min(4vw, 1.5em);}
div.column{flex: auto; overflow-x: auto;}
div.hanging-indent{margin-left: 1.5em; text-indent: -1.5em;}
/* The extra [class] is a hack that increases specificity enough to
override a similar rule in reveal.js */
ul.task-list[class]{list-style: none;}
ul.task-list li input[type="checkbox"] {
font-size: inherit;
width: 0.8em;
margin: 0 0.8em 0.2em -1.6em;
vertical-align: middle;
}
.display.math{display: block; text-align: center; margin: 0.5rem auto;}
/* CSS for citations */
div.csl-bib-body { }
div.csl-entry {
clear: both;
}
.hanging-indent div.csl-entry {
margin-left:2em;
text-indent:-2em;
}
div.csl-left-margin {
min-width:2em;
float:left;
}
div.csl-right-inline {
margin-left:2em;
padding-left:1em;
}
div.csl-indent {
margin-left: 2em;
} </style>
</head>
<body>
<header id="title-block-header">
<h1 class="title">MS MARCO Generative QA Repository Report</h1>
</header>
<nav id="TOC" role="doc-toc">
<ul>
<li><a href="#abstract" id="toc-abstract">Abstract</a></li>
<li><a href="#sec:intro" id="toc-sec:intro">Introduction</a>
<ul>
<li><a href="#background-and-objectives"
id="toc-background-and-objectives">Background and objectives</a></li>
<li><a href="#methodological-commitments"
id="toc-methodological-commitments">Methodological commitments</a></li>
<li><a href="#contributions"
id="toc-contributions">Contributions</a></li>
<li><a href="#roadmap" id="toc-roadmap">Roadmap</a></li>
<li><a href="#repository-status-as-of-2026-07-29"
id="toc-repository-status-as-of-2026-07-29">Repository status as of
2026-07-29</a></li>
</ul></li>
<li><a href="#sec:relwork" id="toc-sec:relwork">Related Work</a></li>
<li><a href="#sec:dataset" id="toc-sec:dataset">Dataset and Exploratory
Data Analysis</a>
<ul>
<li><a href="#datasets-used" id="toc-datasets-used">Datasets
used</a></li>
<li><a href="#query-and-passage-shape"
id="toc-query-and-passage-shape">Query and passage shape</a></li>
<li><a href="#query-type-composition"
id="toc-query-type-composition">Query-type composition</a></li>
<li><a href="#relevance-judgement-sparsity"
id="toc-relevance-judgement-sparsity">Relevance-judgement
sparsity</a></li>
</ul></li>
<li><a href="#sec:method" id="toc-sec:method">Methodology</a>
<ul>
<li><a href="#sec:w2" id="toc-sec:w2">Sparse retrieval (BM25)</a></li>
<li><a href="#sec:w3" id="toc-sec:w3">RAG generation baseline</a></li>
<li><a href="#sec:w4" id="toc-sec:w4">Dense retrieval (sampled)</a></li>
<li><a href="#sec:w5" id="toc-sec:w5">Cross-encoder reranking</a></li>
<li><a href="#sec:w6-method" id="toc-sec:w6-method">Evaluation
layer</a></li>
<li><a href="#sec:w7-method" id="toc-sec:w7-method">Grounding
audit</a></li>
</ul></li>
<li><a href="#sec:results" id="toc-sec:results">Results</a>
<ul>
<li><a href="#sec:w2-results" id="toc-sec:w2-results">BM25 baseline
(full corpus)</a></li>
<li><a href="#sec:w4-results" id="toc-sec:w4-results">Dense vs. BM25 on
the same sample</a></li>
<li><a href="#sec:w4b" id="toc-sec:w4b">Encoder horizontal on the dense
sample</a></li>
<li><a href="#sec:w4a" id="toc-sec:w4a">Density sensitivity of the dense
vs. BM25 gap</a></li>
<li><a href="#sec:w5-dense-rerank"
id="toc-sec:w5-dense-rerank">Cross-encoder reranking on the dense
top-100</a></li>
<li><a href="#sec:trec-dl-results" id="toc-sec:trec-dl-results">TREC-DL
full-corpus external-validity benchmark</a></li>
<li><a href="#sec:beir-results" id="toc-sec:beir-results">BEIR
cross-domain retrieval benchmark</a></li>
<li><a href="#sec:scifact-first-stage"
id="toc-sec:scifact-first-stage">SciFact first-stage diagnosis</a></li>
<li><a href="#sec:scifact-residual-review"
id="toc-sec:scifact-residual-review">SciFact residual first-stage
failure review</a></li>
<li><a href="#sec:cross-dataset-errors"
id="toc-sec:cross-dataset-errors">Cross-dataset first-stage error
analysis</a></li>
<li><a href="#sec:nfcorpus-first-stage"
id="toc-sec:nfcorpus-first-stage">NFCorpus first-stage diagnosis and
bounded query representation</a></li>
<li><a href="#sec:gen-x-retrieval"
id="toc-sec:gen-x-retrieval">Headline: Generation <span
class="math inline">×</span> retrieval source</a></li>
<li><a href="#sec:w5a" id="toc-sec:w5a">First-stage <span
class="math inline">×</span> reranker head-to-head</a></li>
<li><a href="#sec:w6" id="toc-sec:w6">Evaluation layer</a>
<ul>
<li><a href="#sec:w6-bertscore" id="toc-sec:w6-bertscore">Semantic-proxy
BERTScore</a></li>
<li><a href="#sec:w6-taxonomy" id="toc-sec:w6-taxonomy">Regression
failure taxonomy</a></li>
<li><a href="#sec:w6-closure" id="toc-sec:w6-closure">Decoding-budget
closure: <code>mnt=64 </code><span
class="math inline">→</span><code> 128</code></a></li>
<li><a href="#sec:w6-followups" id="toc-sec:w6-followups">Question-form
breakdowns</a></li>
</ul></li>
<li><a href="#sec:w7" id="toc-sec:w7">Grounding audit</a>
<ul>
<li><a href="#lexical-and-3-gram-grounding"
id="toc-lexical-and-3-gram-grounding">Lexical and 3-gram
grounding</a></li>
<li><a href="#sec:w7-nli" id="toc-sec:w7-nli">NLI-entailment
grounding</a></li>
<li><a href="#grounding-leftrightarrow-downstream-correlation"
id="toc-grounding-leftrightarrow-downstream-correlation">Grounding <span
class="math inline">↔︎</span> downstream correlation</a></li>
<li><a href="#low-grounding-case-study"
id="toc-low-grounding-case-study">Low-grounding case study</a></li>
</ul></li>
<li><a href="#sec:w8-triad" id="toc-sec:w8-triad">RAG triad diagnostic
interface</a></li>
<li><a href="#sec:w9-context-packing"
id="toc-sec:w9-context-packing">Context packing and prompt
compression</a></li>
</ul></li>
<li><a href="#sec:discussion" id="toc-sec:discussion">Discussion</a>
<ul>
<li><a href="#sec:discussion-where" id="toc-sec:discussion-where">Where
does the rerank gain come from?</a></li>
<li><a href="#sec:discussion-nli-flip"
id="toc-sec:discussion-nli-flip">The NLI sign flip</a></li>
<li><a href="#sec:plan-vs-actual" id="toc-sec:plan-vs-actual">Plan
vs. actual</a></li>
</ul></li>
<li><a href="#sec:limitations"
id="toc-sec:limitations">Limitations</a></li>
<li><a href="#sec:conclusion" id="toc-sec:conclusion">Conclusion</a>
<ul>
<li><a href="#sec:future-work" id="toc-sec:future-work">Future
work</a></li>
</ul></li>
<li><a href="#acknowledgements"
id="toc-acknowledgements">Acknowledgements</a></li>
<li><a href="#sec:reproducibility"
id="toc-sec:reproducibility">Configuration and reproducibility</a>
<ul>
<li><a href="#one-config-workflow-facade-and-stage-runners"
id="toc-one-config-workflow-facade-and-stage-runners">One config,
workflow facade, and stage runners</a></li>
<li><a href="#manifests" id="toc-manifests">Manifests</a></li>
<li><a href="#input-validation-boundary"
id="toc-input-validation-boundary">Input validation boundary</a></li>
<li><a href="#determinism" id="toc-determinism">Determinism</a></li>
<li><a href="#what-is-and-isnt-committed"
id="toc-what-is-and-isnt-committed">What is and isn’t committed</a></li>
<li><a href="#public-report-and-artifact-surfaces"
id="toc-public-report-and-artifact-surfaces">Public report and artifact
surfaces</a></li>
<li><a href="#engineering-scaffolding"
id="toc-engineering-scaffolding">Engineering scaffolding</a></li>
</ul></li>
<li><a href="#sec:appendix-buckets" id="toc-sec:appendix-buckets">Full
bucket counts and evaluation-layer analysis tables</a></li>
<li><a href="#sec:appendix-glossary"
id="toc-sec:appendix-glossary">Glossary of cross-references</a></li>
</ul>
</nav>
<div class="titlepage">
<hr />
<p><strong>MS MARCO Generative QA</strong></p>
<p><em>A Reproducible Retrieve-Rerank-Generate Pipeline<br />
with Grounding and Triad Audits</em></p>
<p>Repository report - updated 2026-08-11</p>
<hr />
<p><strong>Gioia Zheng</strong></p>
<p><a
href="mailto:gioia.zheng.stud@gmail.com"><code>gioia.zheng.stud@gmail.com</code></a></p>
<table>
<tbody>
<tr>
<td style="text-align: right;"><strong>Core experiment
period</strong></td>
<td style="text-align: left;">2026-03-06 – 2026-05-21</td>
</tr>
<tr>
<td style="text-align: right;"><strong>External validation</strong></td>
<td style="text-align: left;">2026-07-18</td>
</tr>
<tr>
<td style="text-align: right;"><strong>Cross-domain
retrieval</strong></td>
<td style="text-align: left;">2026-07-20</td>
</tr>
<tr>
<td style="text-align: right;"><strong>First-stage
analysis</strong></td>
<td style="text-align: left;">2026-08-11</td>
</tr>
<tr>
<td style="text-align: right;"><strong>Current update</strong></td>
<td style="text-align: left;">2026-08-11</td>
</tr>
<tr>
<td style="text-align: right;"><strong>Scope</strong></td>
<td style="text-align: left;">Batch evaluation, CPU-only; English
passage retrieval + QA</td>
</tr>
</tbody>
</table>
<p><span class="smallcaps">Project surfaces</span><br />
<a
href="https://github.com/GioiaZheng/msmarco-genqa"><code>github.com/GioiaZheng/msmarco-genqa</code></a><br />
<a
href="https://huggingface.co/datasets/GioiaZheng/msmarco-genqa-benchmark-runs"><code>huggingface.co/datasets/GioiaZheng/msmarco-genqa-benchmark-runs</code></a><br />
<a
href="https://gioiazheng.github.io/projects/msmarco-genqa/"><code>gioiazheng.github.io/projects/msmarco-genqa/</code></a></p>
</div>
<h1 class="unnumbered" id="abstract">Abstract</h1>
<p>This report documents a research-engineering project that reproduces
and analyses a complete <em>retrieve–rerank–generate</em>
question-answering pipeline on the MS MARCO Passage Ranking
benchmark <span class="citation"
data-cites="bajaj2016msmarco nguyen2016msmarcoqa">(Bajaj et al. 2016;
Nguyen et al. 2016)</span>. Each stage is implemented from open-source
components on a single laptop, with no GPU, no fine-tuning, and no
proprietary services: BM25 first-stage retrieval <span class="citation"
data-cites="robertson2009bm25 lu2024bm25s">(Robertson and Zaragoza 2009;
Lù 2024)</span> over the 8.8 million-passage corpus; a dense
<code>Sentence-Transformers</code> baseline with FAISS exact
search <span class="citation"
data-cites="reimers2019sbert johnson2019faiss">(Reimers and Gurevych
2019; Johnson et al. 2019)</span> on a qrels-anchored 50k sample; a
cross-encoder reranker <span class="citation"
data-cites="nogueira2019passage wang2020minilm">(Nogueira and Cho 2019;
Wang et al. 2020)</span> over the top-100 candidates; and a frozen
T5-small <span class="citation" data-cites="raffel2020t5">(Raffel et al.
2020)</span> acting as a retrieval-augmented generator <span
class="citation" data-cites="lewis2020rag">(Lewis et al. 2020)</span> on
the top-3 retrieved passages.</p>
<p>The single load-bearing empirical result is a paired comparison on
all 6 980 <em>dev/small</em> queries: replacing BM25 with the
cross-encoder-reranked dense top-3 as the upstream passage source —
without touching the generator — approximately doubles every
surface-form generation metric (Token-F<sub>1</sub> <span
class="math inline">0.197 → 0.368</span>, ROUGE-L <span
class="math inline">0.193 → 0.368</span>), with 95 % paired-bootstrap
confidence intervals <span class="citation"
data-cites="koehn2004bootstrap">(Koehn 2004)</span> strictly above zero
on all four metrics. A DistilBERT-based BERTScore proxy <span
class="citation"
data-cites="zhang2020bertscore sanh2019distilbert">(Zhang et al. 2020;
Sanh et al. 2019)</span> on a 3 000-pair subsample recovers the same
paired <span class="math inline"><em>Δ</em></span> (<span
class="math inline">+0.173</span>), ruling out a surface-form artefact.
A targeted decoding-budget sweep (<span
class="math inline"><code>max_new_tokens</code> = 64 → 128</span>) and a
40-query regression failure taxonomy together <em>falsify</em> the
hypothesis that generator truncation drives the residual regression
bucket: the generator is hitting end-of-sequence naturally on this
prompt format.</p>
<p>A subsequent grounding audit calibrates the headline result. Lexical
content-token grounding (<span class="math inline">0.997</span>) and
3-gram grounding (<span class="math inline">0.99</span>) sit at the
ceiling on both arms, showing that T5-small under the
<code>question: … context: …</code> prompt is performing
<em>extractive</em> rather than truly generative QA: the rerank gain is
almost entirely downstream of retrieval. An NLI-entailment grounding
proxy <span class="citation"
data-cites="he2023debertav3 williams2018mnli">(He et al. 2023; Williams
et al. 2018)</span> produces the only paired metric in the project whose
sign reverses (<span class="math inline"><em>Δ</em> = − 0.145</span>,
95 % CI <span class="math inline">[−0.160, −0.130]</span>), most likely
as an artefact of fragmentary midword-cut outputs that defeat
sentence-level NLI rather than as a true semantic regression.</p>
<p>The report covers, in order: the dataset characterisation, the four
modelling stages, post-hoc evaluation layers (semantic proxy, grounding,
and the current triad diagnostic interface), an honest comparison
between the initial 12-stage project plan and what actually shipped, and
the engineering scaffolding (reproducibility manifests, paired-bootstrap
confidence intervals, deterministic seeds) that lets every number in
this document be regenerated from one command per stage.</p>
<p>This 2026-08-11 update keeps the empirical claims anchored to the
original experiment outputs while bringing the repository description up
to date. Since the experiment window closed, the project has gained an
installable CLI surface, the <code>rag-eval</code> workflow facade,
optional model-stack smoke validation, local experiment tracking, a
lightweight serving wrapper, refreshed CI, secret scanning, a
deterministic RAG triad CLI, stricter input-validation boundaries for
run files / JSONL / serving calls, and a full-corpus TREC-DL 2019/2020
passage benchmark. On the judged topics, cross-encoder reranking raises
MRR@10 by <span class="math inline">+0.332</span> in 2019 and <span
class="math inline">+0.198</span> in 2020 while preserving Recall@100;
graded metrics agree with an independent <code>ir-measures</code>
evaluation within <span
class="math inline">2.22 × 10<sup>−16</sup></span>. The same frozen
reranker also improves MRR@10 on the independent BEIR NFCorpus and
SciFact collections by <span class="math inline">+9.18%</span> and <span
class="math inline">+3.25%</span> relative, respectively. The unchanged
Recall@100 exposes a sharp first-stage ceiling on NFCorpus rather than a
reranker failure.</p>
<p>A fixed-output follow-up makes that NFCorpus ceiling more precise. Of
the 323 test queries, 72 have no judged relevant document in the BM25
top 100; 24 first recover a positive at ranks 101–1,000 and 48 still
miss at depth 1,000. A complete single-reviewer census assigns 67 of the
72 failures to missing source-page context. On the official 102-query
video subset, a predeclared comparison that changes only the query
representation raises Recall@100 from 0.2821 for the BEIR title to
0.3700 for title plus official description (<span
class="math inline"><em>Δ</em> = +0.0880</span>, 95 % paired-bootstrap
CI <span class="math inline">[+0.0535, +0.1265]</span>, <span
class="math inline"><em>p</em> < 0.0002</span>). This is bounded
retrieval evidence: it neither covers the other 221 NFCorpus queries nor
evaluates generation.</p>
<p>The matching cross-dataset error analysis shows why the pipeline
should remain frozen for the next interpretation step. SciFact has much
healthier candidate coverage: Recall@100 is 0.8759 rather than 0.2378,
and 259/300 SciFact queries have complete relevant-document coverage at
depth 100, compared with 19/323 on NFCorpus. NFCorpus still has the
larger residual lexical tail (48 no-hit-at-1,000 queries versus 11 on
SciFact), while both datasets have 24 queries whose first positive hit
appears only at ranks 101–1,000. A bounded review of the 35 residual
SciFact no-hit-at-100 cases shows a different pattern from NFCorpus:
28/35 are best described as scientific claim/evidence formulation
mismatches, 4/35 as lexical competition at the depth cutoff, and 3/35 as
short or broad claims. The evidence separates candidate absence, depth
recoverability, and dataset/query representation effects; it does not by
itself justify a new retrieval architecture.</p>
<p><em>Keywords.</em> MS MARCO; passage retrieval; BM25; dense
retrieval; cross-encoder reranking; retrieval-augmented generation;
BERTScore; NLI grounding; RAG triad; paired bootstrap; extractive
QA.</p>
<h1 id="sec:intro">Introduction</h1>
<h2 id="background-and-objectives">Background and objectives</h2>
<p>The project brief was to build a complete <em>retrieve–generate</em>
question-answering pipeline on the MS MARCO corpus, exploring how each
stage of the pipeline contributes to end-to-end answer quality <span
class="citation" data-cites="bajaj2016msmarco lewis2020rag">(Bajaj et
al. 2016; Lewis et al. 2020)</span>. MS MARCO is a natural target
dataset for this exercise: its queries originate from real Bing user
traffic, its relevance judgements are human-annotated, and its v2.1
Q&A flavour ships human-written answers that allow surface-form
generation evaluation <span class="citation"
data-cites="nguyen2016msmarcoqa">(Nguyen et al. 2016)</span>.</p>
<p>The brief named twelve planned milestones spanning EDA, sparse
retrieval, generation baselines, dense retrieval, reranking, generator
fine-tuning, hallucination mitigation, long-document processing, system
integration and acceleration, and a final write-up. In practice the
project ran through eleven milestones on a single-user, CPU-only Apple
Silicon laptop, with no fine-tuning at any stage. <a
href="#sec:plan-vs-actual" data-reference-type="ref+Label"
data-reference="sec:plan-vs-actual">6.3</a> compares the plan to the
actual deliverables and explains the deviations.</p>
<h2 id="methodological-commitments">Methodological commitments</h2>
<p>Three principles shaped every decision the project made:</p>
<ol>
<li><p><strong>Honest baselines before optimisation.</strong> Every
stage produces a numerically reproducible <em>baseline</em> — BM25 MRR,
generation Token-F<sub>1</sub> on a fixed query set, reranking <span
class="math inline"><em>Δ</em></span>MRR@10 versus a precisely
identified first stage — before any tuning is considered. The 200-query
subsample baselines are clearly labelled as such; the load-bearing
claims are restricted to the full 6 980 <em>dev/small</em>
numbers.</p></li>
<li><p><strong>Paired statistical tests on every comparison.</strong>
When the question is “did X help on top of Y?” the report always answers
it on <em>the same query ids</em>, with a paired bootstrap <span
class="citation" data-cites="koehn2004bootstrap">(Koehn 2004)</span>
reporting a 95 % confidence interval on the per-query <span
class="math inline"><em>Δ</em></span>, not just a point estimate. When
the sample is too small for that — e.g. the 200-query smoke-test scale —
the report says so and labels the number as illustrative.</p></li>
<li><p><strong>Calibrate metrics before celebrating them.</strong>
Surface- form metrics (ROUGE-L, BLEU, EM, Token-F<sub>1</sub>)
under-credit valid paraphrases <span class="citation"
data-cites="lin2004rouge papineni2002bleu">(Lin 2004; Papineni et al.
2002)</span>; a BERTScore semantic proxy <span class="citation"
data-cites="zhang2020bertscore">(Zhang et al. 2020)</span> and a
grounding audit (<a href="#sec:w7" data-reference-type="ref+Label"
data-reference="sec:w7">5.15</a>) check whether the headline gain is
real, an artefact of overlap, or a side-effect of the generator’s output
shape. Two of the three checks confirm the gain; one (the NLI proxy of
<a href="#sec:w7-nli" data-reference-type="ref+Label"
data-reference="sec:w7-nli">5.15.2</a>) flips sign, and the report
devotes <a href="#sec:discussion-nli-flip"
data-reference-type="ref+Label"
data-reference="sec:discussion-nli-flip">6.2</a> to that disagreement
rather than burying it.</p></li>
</ol>
<h2 id="contributions">Contributions</h2>
<p>This report contributes the following:</p>
<ul>
<li><p>A single-machine, CPU-only reproduction of the standard MS MARCO
Passage retrieval <span class="math inline">→</span> reranking <span
class="math inline">→</span> RAG-generation pipeline, with the BM25
first-stage MRR@10 within <span class="math inline">∼ 0.014</span> of
the published Anserini/Lucene reference baseline <span class="citation"
data-cites="lin2021pyserini">(Lin et al. 2021)</span> on the same data
(<a href="#sec:w2" data-reference-type="ref+Label"
data-reference="sec:w2">4.1</a>).</p></li>
<li><p>A <em>paired</em> full-dev comparison of generation quality under
BM25 versus cross-encoder-reranked retrieval (<a
href="#sec:gen-x-retrieval" data-reference-type="ref+Label"
data-reference="sec:gen-x-retrieval">5.12</a>), with 95 %
paired-bootstrap confidence intervals strictly above zero on all four
surface-form metrics and a BERTScore proxy that recovers the same <span
class="math inline"><em>Δ</em></span>.</p></li>
<li><p>A two-step <em>falsification</em> of the most plausible
explanation for the residual regression bucket (<a
href="#sec:w6-closure" data-reference-type="ref+Label"
data-reference="sec:w6-closure">5.14.3</a>): the decoding-budget closure
run and the grounding-ceiling reading together rule out
<code>max_new_tokens</code> as the bottleneck.</p></li>
<li><p>A grounding audit (<a href="#sec:w7"
data-reference-type="ref+Label" data-reference="sec:w7">5.15</a>) that
calibrates the rerank gain as a <em>retrieval</em> gain almost entirely
downstream of passage selection, and isolates a sign-reversing NLI
anomaly (<a href="#sec:w7-nli" data-reference-type="ref+Label"
data-reference="sec:w7-nli">5.15.2</a>) most plausibly attributable to
fragmentary outputs rather than true semantic drift.</p></li>
<li><p>A reproducibility scaffold (per-run manifests with git commit,
dependency hashes, command line, and per-output SHA-256s; deterministic
seeds throughout; a single config file driving all four runners)
sufficient to re-derive every number in this report from one command per
stage on a comparable single-machine setup.</p></li>
<li><p>A maintained repository surface around the experiments:
installable console entry points, the <code>rag-eval</code> workflow
facade, optional model-stack smoke validation, local experiment tracking
with sweep-level JSON/CSV/Markdown summaries, rag-observatory
trace/sweep exports, a lightweight <code>mgq-serve</code> FastAPI
wrapper, CI, secret scanning, and separate reproducibility/security
notes.</p></li>
</ul>
<h2 id="roadmap">Roadmap</h2>
<p><a href="#sec:relwork" data-reference-type="ref+Label"
data-reference="sec:relwork">2</a> surveys the related work that anchors
the design choices. <a href="#sec:dataset"
data-reference-type="ref+Label" data-reference="sec:dataset">3</a>
characterises the MS MARCO subset actually used. <a href="#sec:method"
data-reference-type="ref+Label" data-reference="sec:method">4</a> steps
through the four modelling stages in implementation order. <a
href="#sec:results" data-reference-type="ref+Label"
data-reference="sec:results">5</a> is the central empirical chapter,
presenting the retrieval numbers, the load-bearing paired generation
comparison, the first-stage head-to-head, the evaluation layer, the
grounding audit, and the triad diagnostic interface. <a
href="#sec:discussion" data-reference-type="ref+Label"
data-reference="sec:discussion">6</a> discusses what the combination of
evaluation-layer closure, grounding, and triad diagnostics actually
means about where the gain comes from, treats the NLI sign flip on its
own terms, and compares ship versus plan. <a href="#sec:limitations"
data-reference-type="ref+Label" data-reference="sec:limitations">7</a>
states the limitations. <a href="#sec:conclusion"
data-reference-type="ref+Label" data-reference="sec:conclusion">8</a>
concludes. The question-form breakdown is reported inline with the rest
of the evaluation layer (<a href="#tab:w6b"
data-reference-type="ref+Label" data-reference="tab:w6b">19</a>).
Appendices cover the configuration and reproducibility scaffolding (<a
href="#sec:reproducibility" data-reference-type="ref+Label"
data-reference="sec:reproducibility">9</a>), the full five-way bucket
counts (<a href="#sec:appendix-buckets" data-reference-type="ref+Label"
data-reference="sec:appendix-buckets">10</a>), and a short glossary of
cross-references (<a href="#sec:appendix-glossary"
data-reference-type="ref+Label"
data-reference="sec:appendix-glossary">11</a>).</p>
<h2 id="repository-status-as-of-2026-07-29">Repository status as of
2026-07-29</h2>
<p>The project has moved from a report-centred experiment bundle to a
small research-engineering repository. The headline metrics in this
document remain historical experiment results, but the codebase now has
a clearer operating surface:</p>
<ul>
<li><p><strong>Workflow facade.</strong>
<code>rag-eval run --config configs/baseline.yaml</code> builds or
dry-runs the configured retrieval, reranking, generation,
context-packing, bootstrap, grounding, and triad-evaluation workflow
from one entry point.</p></li>
<li><p><strong>Package and CLI surface.</strong> The project installs
via <code>pip install -e .</code> and exposes stage-level
<code>mgq-*</code> commands plus <code>mgq-serve</code> for local
serving experiments. The current evaluation surface includes
<code>mgq-retrieval-report matrix</code> for same-qid BM25 / dense / RRF
/ reranked comparison tables and <code>mgq-trec-eval</code> for an
independent TREC-compatible metric cross-check, plus
<code>mgq-context-packing-report</code> for packed-vs-plain generation
prompt comparisons, plus <code>mgq-rag-triad</code> for
context-relevance, groundedness, and answer-relevance diagnostics over
existing generation outputs, plus
<code>mgq-export-rag-observatory</code> and
<code>mgq-export-rag-observatory-sweep</code> for exporting selected
query traces and small configuration-sweep bundles to the observability
layer.</p></li>
<li><p><strong>Validation.</strong> Default CI runs fast pytest
coverage, <code>ruff</code>, headline-metric metadata checks, and a
high-confidence secret scan. Heavy model and data checks stay opt-in
through the model-stack smoke and slow/integration test markers. A
committed two-query TREC fixture validates run and qrels parsing,
canonical export, metric scope, and agreement with the optional
<code>ir-measures</code> backend without fetching the full
corpus.</p></li>
<li><p><strong>Reproducibility.</strong> The manifest contract now
records command lines, dependency-file hashes, config hashes, git state,
output hashes, and cache/profile metadata. The documentation separates
quick local checks from heavyweight data/model reproduction.</p></li>
<li><p><strong>TREC-DL external-validity benchmark.</strong> The
full-corpus BM25 and cross-encoder runners cover all 43 judged topics
from 2019 and all 54 from 2020, with outputs isolated by year. Graded
nDCG@10 and threshold-2 MRR@10 / recall pass independent evaluation; <a
href="#sec:trec-dl-results" data-reference-type="ref+Label"
data-reference="sec:trec-dl-results">5.6</a> reports both tracks
separately with query-level lift and failure analysis.</p></li>
<li><p><strong>BEIR cross-domain retrieval benchmark.</strong> NFCorpus
and SciFact use their own corpora and complete test qrels. The unchanged
top-100 cross-encoder improves MRR@10 and nDCG@10 on both collections;
<a href="#sec:beir-results" data-reference-type="ref+Label"
data-reference="sec:beir-results">5.7</a> separates that transfer result
from the unsupported claim of full RAG generalization.</p></li>
<li><p><strong>NFCorpus first-stage diagnosis.</strong> The public BM25
run is now covered by a pinned data-and-metric contract, a complete
72-query no-hit review, and a bounded 102-query query-representation
experiment. The result distinguishes omitted source context from
reranker capacity without changing the architecture.</p></li>
<li><p><strong>SciFact first-stage diagnosis.</strong> The matching
fixed-output check shows a different shape: 265/300 SciFact queries
already have a relevant top-100 candidate and 259/300 have complete
relevant-document coverage at depth 100. The residual 35-query review
points mostly to scientific claim/evidence formulation mismatches,
keeping the NFCorpus source-context ceiling from being overgeneralised
across all BEIR evidence.</p></li>
<li><p><strong>Cross-dataset first-stage error analysis.</strong> The
combined NFCorpus/SciFact diagnosis separates candidate-set absence,
depth-recoverable misses, residual top-1,000 misses, and query/dataset
effects. It supports keeping the pipeline unchanged while deciding
whether a later retrieval-side change is actually justified.</p></li>
<li><p><strong>Public evidence layer.</strong> Fourteen exact ranked
outputs from TREC-DL, BEIR cross-domain, and the NFCorpus video-query
experiment are published as checksum-verified GitHub Release archives
and as a public Hugging Face Dataset. The repository report, dataset
card, and personal project page cross-link the same evidence
boundaries.</p></li>
<li><p><strong>Research backlog.</strong> Follow-up issues now separate
learned sparse retrieval, stronger rerankers, chunking protocols, query
transformation, offline/online indexing, citation-aware generation, and
adaptive retrieval sufficiency checks.</p></li>
</ul>
<h1 id="sec:relwork">Related Work</h1>
<h4 id="the-ms-marco-ecosystem.">The MS MARCO ecosystem.</h4>
<p>MS MARCO is among the largest publicly available collections of
<em>real-user</em> search queries with human-judged relevance and
human-written answers <span class="citation"
data-cites="bajaj2016msmarco nguyen2016msmarcoqa">(Bajaj et al. 2016;
Nguyen et al. 2016)</span>. Its Passage Ranking corpus contains 8.8M
passages with one or two relevance judgements per dev query; the Q&A
v2.1 split layered on top supplies free-form answers that make
end-to-end generation tractable to evaluate. The benchmark has been the
de-facto target for mid-2010s-to-2020s information retrieval research;
published BM25 baselines in the Anserini/Lucene/Pyserini <span
class="citation" data-cites="lin2021pyserini">(Lin et al. 2021)</span>
stack report MRR@10 <span class="math inline"> ≈ 0.184</span> on
<em>dev/small</em>. This project’s BM25 number (MRR@10 <span
class="math inline"> = 0.170</span>, <a href="#sec:w2"
data-reference-type="ref+Label" data-reference="sec:w2">4.1</a>) is
within tokenizer-induced distance of that reference.</p>
<h4 id="sparse-retrieval.">Sparse retrieval.</h4>
<p>BM25 remains the canonical sparse first-stage retriever <span
class="citation" data-cites="robertson2009bm25">(Robertson and Zaragoza
2009)</span>. This project uses <code>bm25s</code> <span
class="citation" data-cites="lu2024bm25s">(Lù 2024)</span>, a
pure-Python re-implementation of the BM25 scoring formula with eager
sparse scoring that achieves Lucene-comparable throughput on commodity
CPUs without requiring a JVM; the trade-off is a different default
tokenizer, which is the main attributable source of the <span
class="math inline">∼ 0.014</span> MRR gap relative to the
Lucene-tokenizer reference baseline.</p>
<h4 id="dense-retrieval.">Dense retrieval.</h4>
<p>Sentence-level dense retrievers <span class="citation"
data-cites="reimers2019sbert">(Reimers and Gurevych 2019)</span> encode
queries and passages into a shared embedding space. ANCE <span
class="citation" data-cites="xiong2021ance">(Xiong et al. 2021)</span>
and subsequent MS MARCO-tuned encoders established that dense retrieval
can substantially outperform BM25 on this benchmark. This project uses a
<em>generic</em> (non-domain-tuned) <code>all-MiniLM-L6-v2</code> <span
class="citation" data-cites="wang2020minilm">(Wang et al. 2020)</span>
encoder as the dense baseline, deliberately leaving the MS MARCO-tuned
variant (<code>msmarco-MiniLM-L6-cos-v5</code>) and a stronger generic
encoder (BGE small <span class="citation" data-cites="xiao2024bge">(Xiao
et al. 2024)</span>) for the same-tier encoder comparison. Exact
similarity search is via FAISS <code>IndexFlatIP</code> <span
class="citation" data-cites="johnson2019faiss">(Johnson et al.
2019)</span> on L2-normalised embeddings (cosine similarity); no
approximate nearest-neighbour index is used, since the 50k-passage
sampled setting fits in memory.</p>
<h4 id="reranking.">Reranking.</h4>
<p>Cross-encoders applied to (query, passage) pairs and trained on
relevance labels — the architecture introduced by Nogueira and Cho <span
class="citation" data-cites="nogueira2019passage">(Nogueira and Cho
2019)</span> — give the highest known single-stage reranking quality on
MS MARCO. The reranker used throughout this project is
<code>cross-encoder/ms-marco-MiniLM-L-6-v2</code>, a small distilled
cross-encoder <span class="citation" data-cites="wang2020minilm">(Wang
et al. 2020)</span> trained on MS MARCO qrels. Reranking depth is fixed
to <span class="math inline"><em>K</em> = 100</span> throughout the
report; the K-sweep ({50, 100, 200}) is queued as the top-k sweep (<a
href="#sec:plan-vs-actual" data-reference-type="ref+Label"
data-reference="sec:plan-vs-actual">6.3</a>).</p>
<h4 id="retrieval-augmented-generation.">Retrieval-augmented
generation.</h4>
<p>RAG <span class="citation" data-cites="lewis2020rag">(Lewis et al.
2020)</span> folds retrieved passages into the prompt of an
encoder–decoder language model. This project’s RAG generator is the
smallest T5 <span class="citation" data-cites="raffel2020t5">(Raffel et
al. 2020)</span> checkpoint (<code>t5-small</code>), used as a frozen
pretrained model with the canonical T5 extractive-QA prompt shape
(<code>question: <q> context: <p1> <p2> <p3></code>;
see <a href="#sec:w3" data-reference-type="ref+Label"
data-reference="sec:w3">4.2</a> for the verbatim form). The choice of
T5-small is deliberate: it caps the absolute generation quality but lets
the entire pipeline run end-to-end on CPU within a few hours per
full-dev sweep, which is what makes the paired statistical comparisons
of <a href="#sec:gen-x-retrieval" data-reference-type="ref+Label"
data-reference="sec:gen-x-retrieval">5.12</a> possible at all.</p>
<h4 id="generation-evaluation.">Generation evaluation.</h4>
<p>Surface-form metrics — ROUGE-L <span class="citation"
data-cites="lin2004rouge">(Lin 2004)</span> and BLEU <span
class="citation" data-cites="papineni2002bleu">(Papineni et al.
2002)</span> — under-credit valid paraphrases. BERTScore <span
class="citation" data-cites="zhang2020bertscore">(Zhang et al.
2020)</span> replaces n-gram overlap with contextualised-embedding
cosine similarity at the token level. This project uses BERTScore with a
DistilBERT <span class="citation" data-cites="sanh2019distilbert">(Sanh
et al. 2019)</span> backbone as a <em>semantic proxy</em> (deliberately
not the canonical <code>roberta-large</code> citation-grade scorer),
reserving the heavier encoder for a follow-up. The grounding audit of <a
href="#sec:w7" data-reference-type="ref+Label"
data-reference="sec:w7">5.15</a> borrows from the
faithfulness-evaluation literature on natural language generation <span
class="citation" data-cites="ji2023hallucination honovich2022true">(Ji
et al. 2023; Honovich et al. 2022)</span>, specifically the practice of
using a small NLI cross-encoder <span class="citation"
data-cites="he2023debertav3 williams2018mnli">(He et al. 2023; Williams
et al. 2018)</span> to score <em>entailment(passages <span
class="math inline">→</span> prediction)</em>.</p>
<h4 id="statistical-testing.">Statistical testing.</h4>
<p>Every paired comparison in this report uses Koehn’s paired
bootstrap <span class="citation" data-cites="koehn2004bootstrap">(Koehn
2004)</span> (10 000 resamples, seed 42, percentile interval);
structural feature comparisons between regression and non-regression
queries use Mann–Whitney U with rank-biserial effect size <span
class="citation" data-cites="mannwhitney1947">(Mann and Whitney
1947)</span>. This is deliberate: with sample sizes in the thousands but
per-query metrics that are highly non-normal (bimodal at 0/1 for
exact-match, heavy-tailed for BLEU on short outputs), only
resampling-based intervals are credible.</p>
<h1 id="sec:dataset">Dataset and Exploratory Data Analysis</h1>
<h2 id="datasets-used">Datasets used</h2>
<p>The project consumes two distinct slices of MS MARCO:</p>
<ul>
<li><p><strong>MS MARCO Passage Ranking</strong> (the retrieval corpus):
8.8M English passages, <span class="math inline">∼ 400</span> bytes
median length. All retrieval baselines (BM25, dense, rerank, and the
generator’s upstream context) index this corpus. Used the official
<code>ir_datasets</code> loader <span class="citation"
data-cites="macavaney2021irdatasets">(MacAvaney et al. 2021)</span> for
both the corpus and the <em>dev/small</em> relevance
judgements.</p></li>
<li><p><strong>MS MARCO Q&A v2.1 validation split</strong>: <span
class="math inline">∼ 101</span>k queries, each with up to ten candidate
passages and one or more human-written reference answers. The EDA uses
the first 5 000 rows of this split; the generation evaluation uses the
references attached to the 6 980 <em>dev/small</em> queries on which
BM25 and the reranker both produce a top-3 (<a
href="#sec:gen-x-retrieval" data-reference-type="ref+Label"
data-reference="sec:gen-x-retrieval">5.12</a>).</p></li>
</ul>
<h2 id="query-and-passage-shape">Query and passage shape</h2>
<p><a href="#fig:query-length,fig:passage-length"
data-reference-type="ref+Label"
data-reference="fig:query-length,fig:passage-length">[fig:query-length,fig:passage-length]</a>
report the empirical length distributions on the EDA 5 000-row
validation sample. Queries are short (median <span
class="math inline">≈ 6</span> tokens; the long tail beyond <span
class="math inline">∼ 15</span> tokens is small), consistent with
web-search behaviour; passages are roughly <span
class="math inline">10×</span> longer (median <span
class="math inline">≈ 50</span> tokens), short enough that the generator
can fit three of them into a T5-small 512-token input window without
per-passage truncation in the common case.</p>
<figure id="fig:query-length" data-latex-placement="H">
<img src="figures/query_length_distribution.png" style="width:78.0%" />
<figcaption>Query-length distribution on the EDA 5 000-row MS MARCO
Q&A v2.1 validation sample. Median <span
class="math inline">≈ 6</span> tokens; the <span
class="math inline">≥ 15</span>-token tail is small. Sets the sizing
target for the RAG prompt budget.</figcaption>
</figure>
<figure id="fig:passage-length" data-latex-placement="H">
<img src="figures/passage_length_distribution.png"
style="width:78.0%" />
<figcaption>Passage-length distribution on the same EDA sample, clipped
at 200 tokens. Median <span class="math inline">≈ 50</span> tokens; the
floor at one sentence is an artefact of MS MARCO’s snippet
extraction.</figcaption>
</figure>
<h2 id="query-type-composition">Query-type composition</h2>
<p>MS MARCO ships a five-way <code>query_type</code> label
(<code>DESCRIPTION</code>, <code>NUMERIC</code>, <code>ENTITY</code>,
<code>LOCATION</code>, <code>PERSON</code>) and a separate (noisier)
free-form answer field. On the 6 980 <em>dev/small</em> queries that the
load-bearing generation comparison uses, the proportions are
<code>DESCRIPTION</code> 53 %, <code>NUMERIC</code> 24 %,
<code>ENTITY</code> 9 %, <code>LOCATION</code> 7 %,
<code>PERSON</code> 7 % (see <a href="#tab:bucket-by-type"
data-reference-type="ref+Label"
data-reference="tab:bucket-by-type">14</a> in <a
href="#sec:gen-x-retrieval" data-reference-type="ref+Label"
data-reference="sec:gen-x-retrieval">5.12</a>).</p>
<p><a href="#fig:query-type,fig:answer-by-type"
data-reference-type="ref+Label"
data-reference="fig:query-type,fig:answer-by-type">[fig:query-type,fig:answer-by-type]</a>
show the EDA distributional read on the validation sample. Two
observations end up shaping later stages:</p>
<ol>
<li><p><code>DESCRIPTION</code> and <code>NUMERIC</code> dominate.
Description answers are long, free-form, and reward paraphrase — which
makes surface-form metrics (ROUGE-L, BLEU, EM) systematically
pessimistic on this class and motivates the BERTScore proxy (<a
href="#sec:w6-bertscore" data-reference-type="ref+Label"
data-reference="sec:w6-bertscore">5.14.1</a>). Numeric answers are short
and rely on lexical surface match in the passage, so the bucket analysis
of <a href="#tab:bucket-by-type" data-reference-type="ref+Label"
data-reference="tab:bucket-by-type">14</a> can use them as a
sanity-check direction (numeric should gain <em>less</em> from
semantic-aware reranking than description does).</p></li>
<li><p>A non-trivial fraction of queries has no answer (or an empty /
“no_answer” string). These must be filtered for any supervised
fine-tuning of the generator; the project does not run SFT, so the
no-answer rows are simply scored through (the generator emits something,
the reference is empty, and the metric for that row is 0). This is
honest but conservative.</p></li>
</ol>
<figure id="fig:query-type" data-latex-placement="H">
<img src="figures/query_type_distribution.png" style="width:78.0%" />
<figcaption>MS MARCO <code>query_type</code> label distribution on the
EDA validation sample. <code>DESCRIPTION</code> dominates;
<code>LOCATION</code> and <code>PERSON</code> are smaller
buckets.</figcaption>
</figure>
<figure id="fig:answer-by-type" data-latex-placement="H">
<img src="figures/answer_type_by_query_type.png" style="width:78.0%" />
<figcaption>Answer-type composition stratified by
<code>query_type</code>. <code>DESCRIPTION</code> queries skew toward
long free-form answers; <code>NUMERIC</code> queries skew toward short /
single-word.</figcaption>
</figure>
<h2 id="relevance-judgement-sparsity">Relevance-judgement sparsity</h2>
<p>A subtle but consequential property of the <em>dev/small</em> qrels:
<span class="math inline">≥ 95 %</span> of dev queries have exactly one
marked-relevant passage in the 8.8M-passage corpus, and <em>none</em>
are negatively labelled. This shapes the dense-retrieval sampling design
(<a href="#sec:w4" data-reference-type="ref+Label"
data-reference="sec:w4">4.3</a>): a uniform random 50k sample from 8.8M
would leave almost no relevant doc in the pool, so the project uses a
<em>qrels-anchored</em> sample in which every dev relevant doc id is
unconditionally included alongside random distractors. The trade-off,
spelled out in <a href="#sec:w4-sampling-caveat"
data-reference-type="ref+Label"
data-reference="sec:w4-sampling-caveat">4.3.0.2</a>, is that absolute
dense/rerank metrics on this sampled setting are upper-bounded by
construction; the <em>within-sample</em> <span
class="math inline"><em>Δ</em></span> between dense and BM25 remains the
meaningful quantity.</p>
<h1 id="sec:method">Methodology</h1>
<p>This section describes the four modelling stages (sparse retrieval,
generation, dense retrieval, and reranking) and the post-hoc analysis
layers (evaluation, grounding, triad diagnostics) in implementation
order. All four modelling stages ship as scripted command-line
entrypoints under <code>experiments/run_*.py</code>, driven by a single
configuration file at <code>configs/baseline.yaml</code>; the evaluation
and grounding layers are analysis scripts under
<code>scripts/*.py</code>, the triad diagnostic is an installable
<code>mgq-rag-triad</code> diagnostic, and context packing is an
installable <code>mgq-context-packing-report</code> comparison over
packed versus plain generation predictions. These layers write JSON /
Markdown summaries beneath the per-analysis directories under
<code>outputs/</code> (the evaluation, grounding,
<code>rag_triad</code>, and <code>context_packing</code> outputs). <a
href="#sec:reproducibility" data-reference-type="ref+Label"
data-reference="sec:reproducibility">9</a> documents the reproducibility
scaffold.</p>
<p><a href="#fig:pipeline" data-reference-type="ref+Label"
data-reference="fig:pipeline">5</a> sketches the end-to-end flow and the
place each stage occupies in it.</p>
<figure id="fig:pipeline" data-latex-placement="H">
<figcaption>End-to-end pipeline. Solid boxes are the four modelling
stages of <a href="#sec:method" data-reference-type="ref+Label"
data-reference="sec:method">4</a>; the dashed box is the post-hoc
evaluation layer (<a href="#sec:w6,sec:w7"
data-reference-type="ref+Label"
data-reference="sec:w6,sec:w7">[sec:w6,sec:w7]</a>), plus the current
triad diagnostic interface, which consumes the already-on-disk paired
generation predictions and does not re-run any modelling stage. BM25 and
dense retrieval are interchangeable first stages feeding into the same
downstream pipeline; the paired generation comparison of <a
href="#sec:gen-x-retrieval" data-reference-type="ref+Label"
data-reference="sec:gen-x-retrieval">5.12</a> switches between
them.</figcaption>
</figure>
<h2 id="sec:w2">Sparse retrieval (BM25)</h2>
<h4 id="setup.">Setup.</h4>
<p>The first stage is BM25 via <code>bm25s</code> <span class="citation"
data-cites="lu2024bm25s">(Lù 2024)</span> on the full 8.8M MS MARCO
Passage corpus, with the canonical Robertson hyperparameters (<span
class="math inline"><em>k</em><sub>1</sub> = 1.5</span>, <span
class="math inline"><em>b</em> = 0.75</span>) and the package-default
English tokenizer and stopword list. Index construction and query
execution run sequentially on a six-core Apple Silicon CPU; the index is
persisted to <code>data/processed/bm25_index_msmarco/</code> (<span
class="math inline">≈ 2.1</span> GiB) and reused across runs. Top-1000
candidates per query are written to disk in TREC <code>run.tsv</code>
format.</p>
<h4 id="retrieve-pass.">Retrieve pass.</h4>
<p>The retrieve pass over 6 980 <em>dev/small</em> queries is
checkpointed every 200 queries (<code>retrieval.chunk_size</code>) into
the same <code>run.tsv</code> target, so an interrupted run can resume