Skip to content

Commit 1c5a6c4

Browse files
artizclaude
andcommitted
all: --ocr-lang option through every surface
The OCR language switch was env-only (DOCLING_RS_OCR_LANG); plumb it through the full option pattern like pages/video_frames: - docling-pdf: OcrLang is now a public enum (parse / from_env); Pipeline::ocr_lang(Option<OcrLang>) + set_ocr_lang for the warm serve instance — a model switch, not pure configuration, so any worker whose cached recognition model was loaded for the other language drops it and lazily reloads (~10 MB model; verified live: ch request then default request on one warm pipeline produce ch-glued and en-spaced output respectively). Worker carries its lang; the one-shot convert_with_options/convert_image_with_options grow an ocr_lang: Option<OcrLang> arg. - docling: DocumentConverter::ocr_lang(str) (unknown value warns and uses the default at convert time, same degradation as the env selector), threaded through StreamSettings into the streaming path; OcrLang re-exported. - CLI: --ocr-lang en|ch (invalid value = usage error, exit 2). - serve: ocr_lang option (multipart field + JSON key + query param), applied per request to the warm pipeline next to set_pages; invalid value → 422. - Python: ocr_lang= kwarg on the native converter (typo → ValueError) and on the docling-shaped wrapper, which also maps ocr_options.lang lists (["english"]→en, ["chinese"]→ch, anything else warns and is ignored). - Node: ocrLang option on ConvertOptions and Pipeline options (invalid → typed Error). Verified against the scanned tiff fixture from JS: ch glues, en spaces, xx throws. - RAG keeps RAG_OCR_LANG (already mapped onto the engine env). Docs: README OCR paragraph (per-surface list), serve API list (also adds the previously undocumented pages line), node/py READMEs, MIGRATION.md PDF row, CLI usage strings. Signed-off-by: artiz <artem.kustikov@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EY5KAiquN4YpVf2PXEQkVT
1 parent c722108 commit 1c5a6c4

15 files changed

Lines changed: 267 additions & 33 deletions

File tree

README.md

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ curl -H 'content-type: application/json' \
125125
```
126126

127127
Options per request: `to=md|json|dclx|chunks`, `strict`, `images=placeholder|embedded`,
128-
`no_ocr`, `no_table_former`, `fetch_images` — as query parameters, multipart
128+
`no_ocr`, `no_table_former`, `pages`, `ocr_lang`, `fetch_images` — as query parameters, multipart
129129
fields, or JSON keys (body wins). Server flags: `--addr`, `--concurrency`,
130130
`--max-body-mb`, `--warmup`, `--no-url-fetch`, `--strict`. A container image
131131
builds from [`crates/docling-serve/Dockerfile`](./crates/docling-serve/Dockerfile)
@@ -759,7 +759,11 @@ env var always wins over the `./models` / `./.pdfium` default.
759759

760760
OCR recognition defaults to the **English** PP-OCRv3 model: the multilingual
761761
`ch_` model reads Latin text with broken word spacing (`Refactorexisting
762-
microservices writtenonJava`-style output on ordinary scans).
762+
microservices writtenonJava`-style output on ordinary scans). The switch
763+
plumbs through every surface — CLI `--ocr-lang en|ch`,
764+
`DocumentConverter::ocr_lang` / `Pipeline::ocr_lang`, serve `ocr_lang`
765+
option, Python `ocr_lang=` kwarg (also mapped from docling-shaped
766+
`ocr_options.lang`), Node `ocrLang` option — or process-wide,
763767
`DOCLING_RS_OCR_LANG=ch` selects the `ch_` pair — that's the model upstream
764768
docling conformance is measured against, and the conformance scripts pin it
765769
themselves; explicit `DOCLING_OCR_REC_ONNX`+`DOCLING_OCR_DICT` (a pair — set

crates/docling-cli/src/main.rs

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
//! The docling.rs counterpart of `docling.cli.main`; `docling-rs serve`
44
//! (with `--features serve`) starts the HTTP conversion API.
55
//!
6-
//! Usage: docling-rs [--strict] [--to md|json] [--pages A-B] [--images MODE] [--fetch-images] [--no-stream] [--no-table-former] [--no-ocr] [--asr-model PRESET] [--video-frames N] [--use-web-browser] [--enrich-picture-classes] [--enrich-code] [--enrich-formula] <input-file>
6+
//! Usage: docling-rs [--strict] [--to md|json] [--pages A-B] [--images MODE] [--fetch-images] [--no-stream] [--no-table-former] [--no-ocr] [--ocr-lang en|ch] [--asr-model PRESET] [--video-frames N] [--use-web-browser] [--enrich-picture-classes] [--enrich-code] [--enrich-formula] <input-file>
77
//! --to md|json output format (default: md). `json` emits docling-core's
88
//! native DoclingDocument JSON (export_to_dict).
99
//! --pages A-B convert only PDF pages A through B (1-based, inclusive;
@@ -94,6 +94,7 @@ fn main() -> ExitCode {
9494
let mut enrich_formula = false;
9595
let mut bench_warm: Option<usize> = None;
9696
let mut pages: Option<(usize, usize)> = None;
97+
let mut ocr_lang: Option<String> = None;
9798
let mut path: Option<String> = None;
9899
let mut args = std::env::args().skip(1);
99100
while let Some(arg) = args.next() {
@@ -130,6 +131,20 @@ fn main() -> ExitCode {
130131
return ExitCode::from(2);
131132
}
132133
},
134+
// OCR recognition language for scanned PDF/image pages: en
135+
// (default; proper Latin word spacing) | ch (the multilingual
136+
// docling-conformance model).
137+
"--ocr-lang" => match args.next() {
138+
Some(v) if matches!(v.trim(), "en" | "ch") => ocr_lang = Some(v),
139+
Some(v) => {
140+
eprintln!("error: --ocr-lang {v:?} is not en|ch");
141+
return ExitCode::from(2);
142+
}
143+
None => {
144+
eprintln!("error: --ocr-lang needs a value (en|ch)");
145+
return ExitCode::from(2);
146+
}
147+
},
133148
// Hidden benchmarking aid: load the PDF/image pipeline once, then time
134149
// N warm conversions (models already loaded), printing the avg seconds
135150
// per conversion to stdout. This is the startup-excluded counterpart to
@@ -168,7 +183,7 @@ fn main() -> ExitCode {
168183
};
169184

170185
let Some(path) = path else {
171-
eprintln!("usage: docling-rs [--strict] [--to md|json] [--images MODE] [--fetch-images] [--no-stream] [--no-table-former] [--no-ocr] [--use-web-browser] <input-file>");
186+
eprintln!("usage: docling-rs [--strict] [--to md|json] [--images MODE] [--fetch-images] [--no-stream] [--no-table-former] [--no-ocr] [--ocr-lang en|ch] [--use-web-browser] <input-file>");
172187
return ExitCode::from(2);
173188
};
174189

@@ -214,6 +229,9 @@ fn main() -> ExitCode {
214229
if let Some((first, last)) = pages {
215230
converter = converter.page_range(first, last);
216231
}
232+
if let Some(lang) = &ocr_lang {
233+
converter = converter.ocr_lang(lang.clone());
234+
}
217235

218236
// Stream Markdown by default: print each chunk as the converter produces it
219237
// (page by page for PDF). Referenced images stream too (#80): each page's

crates/docling-node/README.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -294,6 +294,11 @@ for (const img of res.images) {
294294
295295
JSON output always embeds extracted images as data URIs.
296296
297+
For scanned PDFs/images, `ocrLang: 'en' | 'ch'` picks the OCR recognition
298+
model (`en` is the default — proper Latin word spacing; `ch` is the
299+
multilingual docling-conformance model), and `pages: 'A-B'` converts only that
300+
1-based PDF page window.
301+
297302
## API
298303
299304
### Functions

crates/docling-node/src/lib.rs

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,10 @@ pub struct ConverterOptions {
4040
/// Convert only this PDF page window: `"A-B"` or a single page `"N"`
4141
/// (1-based inclusive — issue #80). Other formats ignore it.
4242
pub pages: Option<String>,
43+
/// OCR recognition language for scanned PDF/image pages: `"en"` (default;
44+
/// proper Latin word spacing) or `"ch"` (the multilingual
45+
/// docling-conformance model). Formats that never OCR ignore it.
46+
pub ocr_lang: Option<String>,
4347
/// Emit cleaner, more conformant Markdown (code-fence languages preserved,
4448
/// no inline-run spacing artifacts) instead of docling's byte-for-byte
4549
/// legacy output. Markdown only. Default `false`.
@@ -80,6 +84,8 @@ pub struct ConvertOptions {
8084
pub video_frames: Option<u32>,
8185
/// PDF page window `"A-B"` (or `"N"`), 1-based inclusive (#80).
8286
pub pages: Option<String>,
87+
/// OCR recognition language for scanned pages: `"en"` (default) | `"ch"`.
88+
pub ocr_lang: Option<String>,
8389
pub allowed_formats: Option<Vec<String>>,
8490
pub to: Option<String>,
8591
pub image_mode: Option<String>,
@@ -135,6 +141,7 @@ struct ConvertConfig {
135141
asr_model: Option<String>,
136142
video_frames: Option<usize>,
137143
page_range: Option<(usize, usize)>,
144+
ocr_lang: Option<String>,
138145
allowed_formats: Option<Vec<InputFormat>>,
139146
to: OutputKind,
140147
image_mode: ImageMode,
@@ -193,13 +200,23 @@ fn build_config(o: ConvertOptions) -> Result<ConvertConfig> {
193200
asr_model: o.asr_model,
194201
video_frames: o.video_frames.map(|n| n as usize),
195202
page_range: parse_pages(o.pages.as_deref())?,
203+
ocr_lang: parse_ocr_lang(o.ocr_lang)?,
196204
allowed_formats: allowed,
197205
to: parse_output_kind(o.to.as_deref())?,
198206
image_mode: parse_image_mode(o.image_mode.as_deref())?,
199207
artifacts_dir: o.artifacts_dir.unwrap_or_else(|| "artifacts".to_string()),
200208
})
201209
}
202210

211+
/// Validate an `ocrLang` option (`"en"`/`"ch"`); an unknown id is an error.
212+
fn parse_ocr_lang(s: Option<String>) -> Result<Option<String>> {
213+
match s {
214+
Some(v) if docling::OcrLang::parse(&v).is_some() => Ok(Some(v)),
215+
Some(v) => Err(Error::from_reason(format!("ocrLang {v:?} is not en|ch"))),
216+
None => Ok(None),
217+
}
218+
}
219+
203220
/// `"A-B"` / `"N"` → the converter's 1-based inclusive page window (#80).
204221
fn parse_pages(s: Option<&str>) -> Result<Option<(usize, usize)>> {
205222
s.map(|v| docling::parse_page_range(v).map_err(|e| Error::from_reason(format!("pages: {e}"))))
@@ -219,9 +236,13 @@ fn build_converter(cfg: &ConvertConfig) -> RsConverter {
219236
Some(max) => base.video_frames(max),
220237
None => base,
221238
};
222-
match cfg.page_range {
239+
let base = match cfg.page_range {
223240
Some((first, last)) => base.page_range(first, last),
224241
None => base,
242+
};
243+
match &cfg.ocr_lang {
244+
Some(lang) => base.ocr_lang(lang.clone()),
245+
None => base,
225246
}
226247
}
227248

@@ -394,6 +415,7 @@ pub struct DocumentConverter {
394415
asr_model: Option<String>,
395416
video_frames: Option<usize>,
396417
page_range: Option<(usize, usize)>,
418+
ocr_lang: Option<String>,
397419
allowed_formats: Option<Vec<InputFormat>>,
398420
}
399421

@@ -416,6 +438,7 @@ impl DocumentConverter {
416438
asr_model: o.asr_model.clone(),
417439
video_frames: o.video_frames.map(|n| n as usize),
418440
page_range: parse_pages(o.pages.as_deref())?,
441+
ocr_lang: parse_ocr_lang(o.ocr_lang.clone())?,
419442
allowed_formats: allowed,
420443
})
421444
}
@@ -428,6 +451,7 @@ impl DocumentConverter {
428451
asr_model: self.asr_model.clone(),
429452
video_frames: self.video_frames,
430453
page_range: self.page_range,
454+
ocr_lang: self.ocr_lang.clone(),
431455
allowed_formats: self.allowed_formats.clone(),
432456
to: parse_output_kind(out.to.as_deref())?,
433457
image_mode: parse_image_mode(out.image_mode.as_deref())?,
@@ -841,6 +865,7 @@ fn output_config(out: Option<OutputOptions>, strict: bool) -> Result<ConvertConf
841865
asr_model: None,
842866
video_frames: None,
843867
page_range: None,
868+
ocr_lang: None,
844869
allowed_formats: None,
845870
to: parse_output_kind(out.to.as_deref())?,
846871
image_mode: parse_image_mode(out.image_mode.as_deref())?,

crates/docling-pdf/src/lib.rs

Lines changed: 57 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ use docling_core::Node;
5353
#[cfg(feature = "ml")]
5454
pub use mets::{convert_mets_gbs, convert_mets_gbs_with_options};
5555
#[cfg(feature = "ml")]
56+
pub use ocr::OcrLang;
5657
pub use pdfium_backend::PdfDocument;
5758
pub use pdfium_backend::{PdfPage, TextCell};
5859

@@ -346,6 +347,8 @@ struct Worker {
346347
/// Skip layout, OCR, and TableFormer; reconstruct text purely from the PDF's
347348
/// embedded text layer. See [`Pipeline::no_ocr`].
348349
no_ocr: bool,
350+
/// Which recognition model [`Self::ocr`] loads. See [`Pipeline::ocr_lang`].
351+
ocr_lang: ocr::OcrLang,
349352
}
350353

351354
#[cfg(feature = "ml")]
@@ -356,6 +359,7 @@ impl Worker {
356359
enrich_slots: (Option<SharedClassifier>, Option<SharedCodeFormula>),
357360
enrich: EnrichmentOptions,
358361
no_ocr: bool,
362+
ocr_lang: ocr::OcrLang,
359363
) -> Result<Self, PdfError> {
360364
Ok(Self {
361365
layout: if no_ocr {
@@ -369,6 +373,7 @@ impl Worker {
369373
code_formula: enrich_slots.1,
370374
enrich,
371375
no_ocr,
376+
ocr_lang,
372377
})
373378
}
374379

@@ -467,7 +472,7 @@ impl Worker {
467472
// No text layer → recognise text from the page image via OCR.
468473
if page.cells.is_empty() {
469474
if self.ocr.is_none() {
470-
self.ocr = Some(ocr::OcrModel::load().map_err(PdfError::Ocr)?);
475+
self.ocr = Some(ocr::OcrModel::load(self.ocr_lang).map_err(PdfError::Ocr)?);
471476
}
472477
let cells = timing::timed("ocr.page", || {
473478
self.ocr
@@ -715,6 +720,8 @@ pub struct Pipeline {
715720
enrich: EnrichmentOptions,
716721
/// 1-based inclusive page window to convert. See [`Pipeline::pages`].
717722
page_range: Option<(usize, usize)>,
723+
/// OCR recognition language. See [`Pipeline::ocr_lang`].
724+
ocr_lang: ocr::OcrLang,
718725
}
719726

720727
#[cfg(feature = "ml")]
@@ -735,6 +742,7 @@ impl Pipeline {
735742
no_ocr: false,
736743
enrich: EnrichmentOptions::default(),
737744
page_range: None,
745+
ocr_lang: ocr::OcrLang::from_env(),
738746
})
739747
}
740748

@@ -757,6 +765,32 @@ impl Pipeline {
757765
self.page_range = range;
758766
}
759767

768+
/// OCR recognition language (see [`OcrLang`]): English by default, `ch`
769+
/// for the multilingual docling-conformance model. `None` keeps the
770+
/// process default (`DOCLING_RS_OCR_LANG`, else English). Set before the
771+
/// first conversion; for a warm pipeline use
772+
/// [`set_ocr_lang`](Self::set_ocr_lang).
773+
pub fn ocr_lang(mut self, lang: Option<ocr::OcrLang>) -> Self {
774+
self.set_ocr_lang(lang);
775+
self
776+
}
777+
778+
/// In-place variant of [`ocr_lang`](Self::ocr_lang) for a long-lived
779+
/// pipeline (docling-serve's warm instance). Unlike the page window this
780+
/// is a *model* switch: any worker whose cached recognition model was
781+
/// loaded for a different language drops it, to be lazily reloaded on the
782+
/// next OCR-needing page (cheap — the rec models are ~10 MB).
783+
pub fn set_ocr_lang(&mut self, lang: Option<ocr::OcrLang>) {
784+
let lang = lang.unwrap_or_else(ocr::OcrLang::from_env);
785+
self.ocr_lang = lang;
786+
for worker in self.primary.iter_mut().chain(self.pool.iter_mut()) {
787+
if worker.ocr_lang != lang {
788+
worker.ocr_lang = lang;
789+
worker.ocr = None;
790+
}
791+
}
792+
}
793+
760794
/// Resolve the configured 1-based window against a page count into the
761795
/// 0-based inclusive form the backend walks, validating it selects at
762796
/// least one existing page.
@@ -854,6 +888,7 @@ impl Pipeline {
854888
self.enrich_slots(),
855889
self.enrich,
856890
self.no_ocr,
891+
self.ocr_lang,
857892
)?);
858893
}
859894
Ok(self.primary.as_mut().unwrap())
@@ -1218,6 +1253,7 @@ impl Pipeline {
12181253
}
12191254
let intra = pdf_intra();
12201255
let no_ocr = self.no_ocr;
1256+
let ocr_lang = self.ocr_lang;
12211257
let enrich = self.enrich;
12221258
let tables = self.tables_slot();
12231259
let enrich_slots = self.enrich_slots();
@@ -1226,7 +1262,9 @@ impl Pipeline {
12261262
.map(|_| {
12271263
let tables = tables.clone();
12281264
let enrich_slots = enrich_slots.clone();
1229-
s.spawn(move || Worker::load(intra, tables, enrich_slots, enrich, no_ocr))
1265+
s.spawn(move || {
1266+
Worker::load(intra, tables, enrich_slots, enrich, no_ocr, ocr_lang)
1267+
})
12301268
})
12311269
.collect();
12321270
handles.into_iter().map(|h| h.join().unwrap()).collect()
@@ -1292,6 +1330,7 @@ pub fn convert(
12921330
false,
12931331
EnrichmentOptions::default(),
12941332
None,
1333+
None,
12951334
)
12961335
}
12971336

@@ -1300,6 +1339,10 @@ pub fn convert(
13001339
/// [`Pipeline::no_table_former`]) and/or layout+OCR+TableFormer entirely (see
13011340
/// [`Pipeline::no_ocr`]), and/or enables the enrichment passes (see
13021341
/// [`Pipeline::enrichments`]).
1342+
// One positional per pipeline switch mirrors the Pipeline builder; growing
1343+
// past clippy's arity cap is the price of keeping this one-shot signature
1344+
// stable-ish instead of churning callers into an options struct mid-series.
1345+
#[allow(clippy::too_many_arguments)]
13031346
pub fn convert_with_options(
13041347
bytes: &[u8],
13051348
password: Option<&str>,
@@ -1308,19 +1351,28 @@ pub fn convert_with_options(
13081351
no_ocr: bool,
13091352
enrich: EnrichmentOptions,
13101353
pages: Option<(usize, usize)>,
1354+
ocr_lang: Option<OcrLang>,
13111355
) -> Result<DoclingDocument, PdfError> {
13121356
Pipeline::new()?
13131357
.no_table_former(no_table_former)
13141358
.no_ocr(no_ocr)
13151359
.enrichments(enrich)
13161360
.pages(pages)
1361+
.ocr_lang(ocr_lang)
13171362
.convert(bytes, password, name)
13181363
}
13191364

13201365
#[cfg(feature = "ml")]
13211366
/// Convenience one-shot image conversion (loads the pipeline per call).
13221367
pub fn convert_image(bytes: &[u8], name: &str) -> Result<DoclingDocument, PdfError> {
1323-
convert_image_with_options(bytes, name, false, false, EnrichmentOptions::default())
1368+
convert_image_with_options(
1369+
bytes,
1370+
name,
1371+
false,
1372+
false,
1373+
EnrichmentOptions::default(),
1374+
None,
1375+
)
13241376
}
13251377

13261378
#[cfg(feature = "ml")]
@@ -1333,11 +1385,13 @@ pub fn convert_image_with_options(
13331385
no_table_former: bool,
13341386
no_ocr: bool,
13351387
enrich: EnrichmentOptions,
1388+
ocr_lang: Option<OcrLang>,
13361389
) -> Result<DoclingDocument, PdfError> {
13371390
Pipeline::new()?
13381391
.no_table_former(no_table_former)
13391392
.no_ocr(no_ocr)
13401393
.enrichments(enrich)
1394+
.ocr_lang(ocr_lang)
13411395
.convert_image(bytes, name)
13421396
}
13431397

0 commit comments

Comments
 (0)