Skip to content

Commit 1ad8e48

Browse files
artizclaude
andcommitted
vlm: lock DocTags→Markdown parity for picture_classification (#153)
picture_classification scored 1.3% similarity in the live VLM corpus run. Root-causing it deterministically: our DocTags parser reproduces docling's groundtruth Markdown for that fixture byte-for-byte (pictures with captions before the image placeholder, multi-page <page_break>, reading order), so the divergence is live-model / render variance — which the harness itself flags as expected ("triage as render, not parser") — not a defect on our side. Add an end-to-end regression test through convert_vlm that feeds the model's exact DocTags for the document and asserts the Markdown matches the groundtruth, so our side of the comparison stays provably correct. Image leg, so it needs no pdfium and runs in CI everywhere. Refs #153 Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
1 parent 5522d02 commit 1ad8e48

1 file changed

Lines changed: 32 additions & 0 deletions

File tree

crates/docling/tests/vlm.rs

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -169,6 +169,38 @@ fn vlm_parses_doctags_answers() {
169169
assert!(md.contains("| H"), "md: {md:?}");
170170
}
171171

172+
/// Conformance lock for #153: the `picture_classification` corpus fixture
173+
/// scored 1.3% similarity in the live VLM run — a divergence that (per the
174+
/// harness's own note) is render/model variance, *not* a parser defect. Feed
175+
/// the model's exact DocTags for that document through `convert_vlm` and assert
176+
/// the Markdown matches docling's groundtruth byte-for-byte, so our side of the
177+
/// comparison stays provably correct (pictures + captions before the image
178+
/// placeholder, multi-page `<page_break>`, reading order). Image leg → no
179+
/// pdfium, runs everywhere.
180+
#[test]
181+
fn vlm_doctags_matches_groundtruth_picture_classification() {
182+
let doctags = std::fs::read_to_string(
183+
repo_root().join("tests/data/pdf/groundtruth/picture_classification.doctags.txt"),
184+
)
185+
.expect("doctags fixture");
186+
let want = std::fs::read_to_string(
187+
repo_root().join("tests/data/pdf/groundtruth/picture_classification.md"),
188+
)
189+
.expect("markdown groundtruth");
190+
// The model emits one DocTags blob per document (root wrapper + internal
191+
// <page_break>); the image leg is a single request → single answer.
192+
let (endpoint, served, handle) = mock_openai(vec![doctags]);
193+
let source = SourceDocument::from_bytes("page.png", InputFormat::Image, image_bytes());
194+
let doc = convert_vlm(&source, &opts(endpoint)).expect("vlm conversion");
195+
handle.join().expect("mock server");
196+
assert_eq!(served.load(Ordering::SeqCst), 1);
197+
assert_eq!(
198+
doc.export_to_markdown().trim(),
199+
want.trim(),
200+
"VLM DocTags → Markdown must match docling's groundtruth for picture_classification"
201+
);
202+
}
203+
172204
#[test]
173205
fn vlm_rejects_non_visual_formats() {
174206
let source = SourceDocument::from_bytes("x.md", InputFormat::Md, b"# hi".to_vec());

0 commit comments

Comments
 (0)