pdf: VLM pipeline — remote OpenAI-compatible vision endpoint (#77)

docling's VlmPipeline in its remote (ApiVlmOptions) form: render each
PDF page via pdfium, POST it (base64 PNG data URI) with docling's
"Convert this page to docling." prompt to any OpenAI-compatible
chat/completions endpoint (LM Studio / Ollama / vLLM / hosted), and
parse the returned markup with the existing tolerant DocLang reader —
per-page fragments are normalized (fence stripping, <doclang>/<doctag>
root unwrapping) and stitched under one root, so the whole document goes
through backend::doclang exactly like a .dclg file. Image inputs are
sent as-is (they are their own page). No model code compiled in; local
ONNX inference of a docling VLM stays a later enhancement.

New `vlm` cargo feature on the docling crate (in defaults — it rides the
ureq + pdf machinery the default build already carries; the wasm/
no-default builds are untouched). docling::vlm::{VlmOptions,
convert_vlm}; endpoint accepts a /v1 base or a full …/chat/completions
URL; DOCLING_RS_VLM_{ENDPOINT,MODEL,PROMPT,API_KEY} env fallbacks.
Transport errors and 408/429/5xx retry 4 attempts with 2s/4s/8s backoff;
any page that still fails fails the conversion — no silently dropped
pages. temperature 0, max_tokens 8192. HTTP goes over ureq, the crate's
existing blocking client (the issue sketch said reqwest, but the
converter is synchronous and fetch-images already pins ureq — one HTTP
stack, no async runtime).

CLI: --pipeline standard|vlm, --vlm-endpoint URL, --vlm-model NAME.
--pages composes (only the window renders and ships); --to md|json|dclx|
chunks and --strict work via the shared buffered-output tail (factored
out as output_document). Verified end to end against a local mock
server: two-page conversion, per-page requests, markdown out; a missing
endpoint errors with the flag/env hint.

Tests: response-normalization unit tests; a mock OpenAI server e2e that
renders two real pages (skips cleanly without pdfium), asserts the wire
shape (path, model, prompt, data URI) and stitches differently-wrapped
answers; an image-input leg that runs everywhere; a non-visual-format
rejection. Docs: README "VLM pipeline (remote endpoint)" section —
explicitly infrastructure, not a conformance claim: corpus comparison
against docling's VLM output still needs a live endpoint —
MIGRATION.md §5 + at-a-glance row.

Refs #77

Signed-off-by: artiz <artem.kustikov@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT

Author: artiz

README.md

@@ -376,6 +376,34 @@ by a wide margin, but a scanned/image-only PDF (no embedded text layer) comes
 back empty rather than erroring, so a caller can detect that and re-convert
 without the flag.
 
+### VLM pipeline (remote endpoint)
+
+`--pipeline vlm` (issue #77) replaces the whole discriminative ML stack with a
+Vision Language Model: each PDF page is rendered (pdfium) and sent to any
+**OpenAI-compatible** vision endpoint — LM Studio, Ollama, vLLM, or a hosted
+service — with docling's page-conversion prompt; the returned DocLang markup
+is parsed by the same reader that `.dclg`/`.dclx` inputs use. An image input
+is sent as-is (it is its own page). No ONNX models load at all; local
+in-process VLM inference is a possible later enhancement.
+
+```bash
+docling-rs --pipeline vlm \
+  --vlm-endpoint http://localhost:11434/v1 \
+  --vlm-model granite-docling \
+  paper.pdf
+```
+
+`--vlm-endpoint` takes the server's `/v1` base or the full
+`…/chat/completions` URL; `DOCLING_RS_VLM_ENDPOINT` / `DOCLING_RS_VLM_MODEL` /
+`DOCLING_RS_VLM_PROMPT` / `DOCLING_RS_VLM_API_KEY` (Bearer token) are the env
+equivalents. `--pages A-B` composes (only the window's pages are rendered and
+sent), and `--to md|json|dclx|chunks` plus `--strict` work as usual. Transient
+endpoint failures (timeouts, 408/429, 5xx) retry with exponential backoff;
+a page that still fails fails the conversion — no silently dropped pages.
+Output quality is entirely the model's: conversion of the PDF corpus against
+docling's own VLM pipeline output hasn't been measured yet (it needs a live
+endpoint), so treat this as infrastructure, not a conformance claim.
+
 ### Headless-browser HTML pre-render (optional)
 
 Almost everything in the HTML backend is pure Rust, but one thing a static

crates/docling-cli/src/main.rs

@@ -3,7 +3,7 @@
 //! The docling.rs counterpart of `docling.cli.main`; `docling-rs serve`
 //! (with `--features serve`) starts the HTTP conversion API.
 //!
-//! Usage: docling-rs [--strict] [--to md|json] [--pages A-B] [--images MODE] [--fetch-images] [--no-stream] [--no-table-former] [--no-ocr] [--ocr-lang en|ch] [--asr-model PRESET] [--video-frames N] [--use-web-browser] [--enrich-picture-classes] [--enrich-code] [--enrich-formula] <input-file>
+//! Usage: docling-rs [--strict] [--to md|json] [--pages A-B] [--images MODE] [--fetch-images] [--no-stream] [--no-table-former] [--no-ocr] [--ocr-lang en|ch] [--pipeline standard|vlm] [--vlm-endpoint URL] [--vlm-model NAME] [--asr-model PRESET] [--video-frames N] [--use-web-browser] [--enrich-picture-classes] [--enrich-code] [--enrich-formula] <input-file>
 //!   --to md|json       output format (default: md). `json` emits docling-core's
 //!                      native DoclingDocument JSON (export_to_dict).
 //!   --pages A-B        convert only PDF pages A through B (1-based, inclusive;
@@ -95,6 +95,9 @@ fn main() -> ExitCode {
     let mut bench_warm: Option<usize> = None;
     let mut pages: Option<(usize, usize)> = None;
     let mut ocr_lang: Option<String> = None;
+    let mut pipeline: Option<String> = None;
+    let mut vlm_endpoint: Option<String> = None;
+    let mut vlm_model: Option<String> = None;
     let mut path: Option<String> = None;
     let mut args = std::env::args().skip(1);
     while let Some(arg) = args.next() {
@@ -145,6 +148,22 @@ fn main() -> ExitCode {
                     return ExitCode::from(2);
                 }
             },
+            // Pipeline selection (#77): `standard` (default, the ML stack) or
+            // `vlm` — render pages and convert them through a remote
+            // OpenAI-compatible vision endpoint returning DocLang.
+            "--pipeline" => match args.next() {
+                Some(v) if matches!(v.trim(), "standard" | "vlm") => pipeline = Some(v),
+                Some(v) => {
+                    eprintln!("error: --pipeline {v:?} is not standard|vlm");
+                    return ExitCode::from(2);
+                }
+                None => {
+                    eprintln!("error: --pipeline needs a value (standard|vlm)");
+                    return ExitCode::from(2);
+                }
+            },
+            "--vlm-endpoint" => vlm_endpoint = args.next(),
+            "--vlm-model" => vlm_model = args.next(),
             // Hidden benchmarking aid: load the PDF/image pipeline once, then time
             // N warm conversions (models already loaded), printing the avg seconds
             // per conversion to stdout. This is the startup-excluded counterpart to
@@ -213,6 +232,29 @@ fn main() -> ExitCode {
         };
     }
 
+    // #77: the remote-VLM pipeline replaces the whole ML stack — convert,
+    // then fall through to the regular output selection (md/json/dclx/chunks
+    // all work; there is no page-streaming, the endpoint is the bottleneck).
+    if pipeline.as_deref() == Some("vlm") {
+        let mut opts = match docling::vlm::VlmOptions::resolve(vlm_endpoint, vlm_model) {
+            Ok(o) => o,
+            Err(e) => {
+                eprintln!("error: {e}");
+                return ExitCode::from(2);
+            }
+        };
+        opts.page_range = pages;
+        let mut document = match docling::vlm::convert_vlm(&source, &opts) {
+            Ok(doc) => doc,
+            Err(e) => {
+                eprintln!("error: {e}");
+                return ExitCode::FAILURE;
+            }
+        };
+        document.strict_markdown = strict;
+        return output_document(document, &to, image_mode, &path);
+    }
+
     let mut converter = DocumentConverter::new()
         .strict(strict)
         .asr_model(asr_model.clone())
@@ -281,7 +323,17 @@ fn main() -> ExitCode {
             return ExitCode::FAILURE;
         }
     };
+    output_document(document, &to, image_mode, &path)
+}
 
+/// The buffered output tail shared by the standard (non-streaming) and VLM
+/// paths: `--to` selection, image sidecars, exit code.
+fn output_document(
+    document: docling::DoclingDocument,
+    to: &str,
+    image_mode: ImageMode,
+    path: &str,
+) -> ExitCode {
     if to == "json" {
         println!("{}", document.export_to_json());
         return ExitCode::SUCCESS;

crates/docling/Cargo.toml

@@ -20,7 +20,7 @@ exclude = ["tests", "benches"]
 # client are feature-sliced so `--no-default-features` leaves the pure-Rust
 # declarative converters (DOCX/HTML/MD/XLSX/PPTX/… → md/json) — a set that
 # compiles for wasm32-unknown-unknown (issue #79, see crates/docling-wasm).
-default = ["pdf", "asr", "fetch-images"]
+default = ["pdf", "asr", "fetch-images", "vlm"]
 # PDF/image/METS-GBS ML pipeline (pdfium + ONNX Runtime via docling-pdf).
 pdf = ["dep:docling-pdf", "docling-pdf/ml"]
 # Text-layer-only PDF conversion (docling-pdf's pure-Rust parser, no pdfium/
@@ -30,6 +30,11 @@ pdf-text = ["dep:docling-pdf"]
 asr = ["dep:docling-asr"]
 # Blocking HTTP fetch of remote <img src> images (DocumentConverter::fetch_images).
 fetch-images = ["dep:ureq", "dep:url"]
+# Remote VLM pipeline (#77): render PDF pages via pdfium and convert them
+# through an OpenAI-compatible vision endpoint (LM Studio / Ollama / vLLM /
+# hosted), parsing the returned DocLang with the existing reader. Rides on the
+# crate's existing blocking HTTP client; no model code compiled in.
+vlm = ["pdf", "dep:ureq"]
 # Optional headless-browser HTML pre-render (the `--use-web-browser` flag /
 # `DocumentConverter::use_web_browser`). Off by default so the standard build
 # stays pure-Rust with no browser/runtime dependency; enable with

crates/docling/src/backend/mod.rs

@@ -29,7 +29,7 @@ mod cfb;
 mod csv;
 mod deepseek;
 mod doc;
-mod doclang;
+pub(crate) mod doclang;
 mod docling_json;
 mod docx;
 mod email;

crates/docling/src/lib.rs

@@ -32,6 +32,8 @@ mod result;
 mod source;
 #[cfg(feature = "pdf")]
 mod stream;
+#[cfg(feature = "vlm")]
+pub mod vlm;
 
 pub mod backend;
 #[cfg(feature = "asr")]