Skip to content

Commit 7d565a9

Browse files
artizclaude
andcommitted
feat(markdown): strict mode tightens punctuation spacing
Strict mode already diverges from docling for readability (underscore un-escaping, code-fence languages). Extend it to tighten stray spaces around punctuation: `[ 37 , 36 ]` → `[37, 36]`, `( x )` → `(x)`, `*c* .` → `*c*.`. This cleans up both the PDF backend's glyph-split citation spacing and the space the legacy emphasis serialization leaves before punctuation. Default/legacy output is untouched (still byte-for-byte with docling, which keeps those spaces). Only inline text nodes are affected — code blocks and table cells are left alone. Regenerated the strict regression fixtures; added a unit test pinning legacy-spacing vs strict-tightened. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent fd276c6 commit 7d565a9

20 files changed

Lines changed: 173 additions & 154 deletions

crates/fleischwolf-core/src/markdown.rs

Lines changed: 27 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -61,16 +61,25 @@ pub fn to_markdown_images(
6161
(md, ctx.artifacts)
6262
}
6363

64-
/// In `strict` mode, undo the legacy `\_` underscore escaping the backends bake
65-
/// into inline text. Legacy output keeps `\_` (byte-for-byte with docling, which
66-
/// escapes underscores); strict prefers literal `_` for readability. Only inline
67-
/// text nodes are escaped — code blocks and table cells are left alone.
64+
/// In `strict` mode, rewrite inline text for readability rather than byte-for-byte
65+
/// docling fidelity: undo the legacy `\_` underscore escaping, and tighten stray
66+
/// spaces around punctuation (`[ 37 , 36 ]` → `[37, 36]`, `( x )` → `(x)`). This
67+
/// cleans up both the PDF backend's glyph-split spacing and the space the legacy
68+
/// emphasis serialization leaves before punctuation (`*a* ,` → `*a*,`).
69+
/// Legacy/default output keeps docling's spacing untouched. Only inline text
70+
/// nodes pass through here — code blocks and table cells are left alone.
6871
fn strict_text(text: &str, strict: bool) -> String {
69-
if strict {
70-
text.replace("\\_", "_")
71-
} else {
72-
text.to_string()
72+
if !strict {
73+
return text.to_string();
7374
}
75+
text.replace("\\_", "_")
76+
.replace(" ,", ",")
77+
.replace(" .", ".")
78+
.replace(" ;", ";")
79+
.replace(" )", ")")
80+
.replace("( ", "(")
81+
.replace(" ]", "]")
82+
.replace("[ ", "[")
7483
}
7584

7685
fn render(nodes: &[Node], blocks: &mut Vec<String>, ctx: &mut Ctx) {
@@ -363,4 +372,14 @@ mod tests {
363372
// Strict prefers literal underscores (Rust-only readability mode).
364373
assert_eq!(doc.export_to_markdown_with(true), "# a_b\n\nx_y\n\n- i_j\n");
365374
}
375+
376+
#[test]
377+
fn strict_tightens_punctuation_spacing_legacy_keeps_it() {
378+
let mut doc = DoclingDocument::new("t");
379+
doc.add_paragraph("see [ 37 , 36 ] and ( x ) .");
380+
// Legacy keeps docling's spacing byte-for-byte.
381+
assert_eq!(doc.export_to_markdown(), "see [ 37 , 36 ] and ( x ) .\n");
382+
// Strict tightens punctuation for readable Markdown.
383+
assert_eq!(doc.export_to_markdown_with(true), "see [37, 36] and (x).\n");
384+
}
366385
}

crates/fleischwolf/tests/data/docx/expected/docx_checkboxes.docx.strict.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,4 @@ Tasks before release:
1414

1515
- [x] Implementation
1616

17-
- [ ] Documentation of the task with a very long text that goes beyond the maximum length of a single line.
17+
- [] Documentation of the task with a very long text that goes beyond the maximum length of a single line.

crates/fleischwolf/tests/data/docx/expected/docx_rich_cells.docx.strict.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ Before table
1515
| Simple cell upper left | Simple cell with **bold** and *italic* text |
1616
| A B C Cell 1 Cell 2 Cell 3 | Rich cell A nested table A B C Cell 1 Cell 2 Cell 3 |
1717

18-
After table with **bold** , underline , ~~strikethrough~~ , and *italic* formatting
18+
After table with **bold**, underline, ~~strikethrough~~, and *italic* formatting
1919

2020
### Table with pictures
2121

crates/fleischwolf/tests/data/docx/expected/equations.docx.strict.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
This is a word document and this is an inline equation: $A= \pi r^{2}$ .
1+
This is a word document and this is an inline equation: $A= \pi r^{2}$.
22

33
First item with inline equation: $A= \pi r^{2}$ is the area formula.
44

@@ -18,7 +18,7 @@ $$f\left(x\right)=a_{0}+\sum_{n=1}^{ \infty }\left(a_{n}\cos(\frac{n \pi x}{L})+
1818

1919
This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text. This is text.
2020

21-
This is a word document and this is an inline equation: $A= \pi r^{2}$ . If instead, I want an equation by line, I can do this:
21+
This is a word document and this is an inline equation: $A= \pi r^{2}$. If instead, I want an equation by line, I can do this:
2222

2323
$$\left(x+a\right)^{n}=\sum_{k=0}^{n}\left(\genfrac{}{}{0pt}{}{n}{k}\right)x^{k}a^{n-k}$$
2424

@@ -32,7 +32,7 @@ This is text. This is text. This is text. This is text. This is text. This is te
3232

3333
This is a word document and these are inline equations: $N_{s}^{H}$ / $N_{s}^{P}$ ​. If instead, I want an equation by line, I can do this:
3434

35-
$$e^{x}=1+\frac{x}{1!}+\frac{x^{2}}{2!}+\frac{x^{3}}{3!}+ \text{ \textellipsis } , - \infty <x< \infty$$
35+
$$e^{x}=1+\frac{x}{1!}+\frac{x^{2}}{2!}+\frac{x^{3}}{3!}+ \text{ \textellipsis }, - \infty <x< \infty$$
3636

3737
And that is an equation by itself. Cheers!
3838

@@ -60,7 +60,7 @@ $$\oiiint_{C}^{}\frac{1}{z}dz$$
6060

6161
Operators used with limits:
6262

63-
$$argmax_{ \epsilon }f(x), \lim_{n}{\left(1+\frac{1}{n}\right)}^{n} , \max_{0 \leq x \leq 1}xe^{-x^{2}}, unsupported_{n}{\left(1+\frac{1}{n}\right)}^{n}$$
63+
$$argmax_{ \epsilon }f(x), \lim_{n}{\left(1+\frac{1}{n}\right)}^{n}, \max_{0 \leq x \leq 1}xe^{-x^{2}}, unsupported_{n}{\left(1+\frac{1}{n}\right)}^{n}$$
6464

6565
Equations with the OMML group character object:
6666

crates/fleischwolf/tests/data/docx/expected/textbox.docx.strict.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ Upon receiving a report from the kindergarten, conduct a preliminary assessment
3232
Department of Education:
3333
Collaborate with the Health Bureau in conducting epidemiological investigations and assist Health Bureau personnel in implementing necessary epidemic prevention measures at all school levels.
3434

35-
The Health Bureau will handle **reporting and specimen collection** .
35+
The Health Bureau will handle **reporting and specimen collection**.
3636

3737
**Whether the epidemic has eased.**
3838

crates/fleischwolf/tests/data/epub/expected/epub_purvis_poetry.epub.strict.md

Lines changed: 16 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Poetry
22

3-
By **Sarah Louisa Forten Purvis** .
3+
By **Sarah Louisa Forten Purvis**.
44

55
<!-- image -->
66

@@ -10,13 +10,13 @@ The Standard Ebooks logo.
1010

1111
<!-- image -->
1212

13-
This ebook is the product of many hours of hard work by volunteers for [Standard Ebooks](https://standardebooks.org/) , and builds on the hard work of other literature lovers made possible by the public domain.
13+
This ebook is the product of many hours of hard work by volunteers for [Standard Ebooks](https://standardebooks.org/), and builds on the hard work of other literature lovers made possible by the public domain.
1414

15-
This particular ebook is based on digital scans from the [Internet Archive](https://standardebooks.org/ebooks/sarah-louisa-forten-purvis/poetry#page-scans) .
15+
This particular ebook is based on digital scans from the [Internet Archive](https://standardebooks.org/ebooks/sarah-louisa-forten-purvis/poetry#page-scans).
1616

17-
The source text and artwork in this ebook are believed to be in the United States public domain; that is, they are believed to be free of copyright restrictions in the United States. They may still be copyrighted in other countries, so users located outside of the United States must check their local laws before using this ebook. The creators of, and contributors to, this ebook dedicate their contributions to the worldwide public domain via the terms in the [CC0 1.0 Universal Public Domain Dedication](https://creativecommons.org/publicdomain/zero/1.0/) . For full license information, see the [Uncopyright](uncopyright.xhtml) at the end of this ebook.
17+
The source text and artwork in this ebook are believed to be in the United States public domain; that is, they are believed to be free of copyright restrictions in the United States. They may still be copyrighted in other countries, so users located outside of the United States must check their local laws before using this ebook. The creators of, and contributors to, this ebook dedicate their contributions to the worldwide public domain via the terms in the [CC0 1.0 Universal Public Domain Dedication](https://creativecommons.org/publicdomain/zero/1.0/). For full license information, see the [Uncopyright](uncopyright.xhtml) at the end of this ebook.
1818

19-
Standard Ebooks is a volunteer-driven project that produces ebook editions of public domain literature using modern typography, technology, and editorial standards, and distributes them free of cost. You can download this and other ebooks carefully produced for true book lovers at [standardebooks.org](https://standardebooks.org/) .
19+
Standard Ebooks is a volunteer-driven project that produces ebook editions of public domain literature using modern typography, technology, and editorial standards, and distributes them free of cost. You can download this and other ebooks carefully produced for true book lovers at [standardebooks.org](https://standardebooks.org/).
2020

2121
## The Grave of the Slave
2222

@@ -200,7 +200,7 @@ This bursting heart relief?
200200

201201
There's nothing left for me to love
202202
This earth holds nothing dear,
203-
Since *he* , my sweet-my gentle one,
203+
Since *he*, my sweet-my gentle one,
204204
Is now no longer here.
205205

206206
My poor fond heart had counted on
@@ -465,34 +465,34 @@ The Standard Ebooks logo.
465465

466466
*Poetry*
467467
was compiled from poems published between 1831 and 1836 by
468-
[Sarah Louisa Forten Purvis](https://en.wikipedia.org/wiki/Sarah_Louisa_Forten_Purvis) .
468+
[Sarah Louisa Forten Purvis](https://en.wikipedia.org/wiki/Sarah_Louisa_Forten_Purvis).
469469

470470
This ebook was transcribed and produced for
471471
[Standard Ebooks](https://standardebooks.org/)
472472
by
473-
[Weijia Cheng](https://weijiarhymeswith.asia/) ,
473+
[Weijia Cheng](https://weijiarhymeswith.asia/),
474474
and is based on digital scans from the
475-
[Internet Archive](https://standardebooks.org/ebooks/sarah-louisa-forten-purvis/poetry#page-scans) .
475+
[Internet Archive](https://standardebooks.org/ebooks/sarah-louisa-forten-purvis/poetry#page-scans).
476476

477477
The cover page is adapted from
478-
*Sunday Morning* ,
478+
*Sunday Morning*,
479479
a painting completed in 1877 by
480-
[Thomas Waterman Wood](https://en.wikipedia.org/wiki/Thomas_Waterman_Wood) .
480+
[Thomas Waterman Wood](https://en.wikipedia.org/wiki/Thomas_Waterman_Wood).
481481
The cover and title pages feature the
482482
**League Spartan** and **Sorts Mill Goudy**
483483
typefaces created in 2014 and 2009 by
484-
[The League of Moveable Type](https://www.theleagueofmoveabletype.com/) .
484+
[The League of Moveable Type](https://www.theleagueofmoveabletype.com/).
485485

486486
This edition was released on
487487
**May 21, 2026, 6:29** **p.m.**
488488
and is based on
489-
**revision 936122b** .
489+
**revision 936122b**.
490490
The first edition of this ebook was released on
491491
December 6, 2023, 8:52 p.m.
492492
You can check for updates to this ebook, view its revision history, or download it for different ereading systems at
493-
[standardebooks.org/ebooks/sarah-louisa-forten-purvis/poetry](https://standardebooks.org/ebooks/sarah-louisa-forten-purvis/poetry) .
493+
[standardebooks.org/ebooks/sarah-louisa-forten-purvis/poetry](https://standardebooks.org/ebooks/sarah-louisa-forten-purvis/poetry).
494494

495-
The volunteer-driven Standard Ebooks project relies on readers like you to submit typos, corrections, and other improvements. Anyone can contribute at [standardebooks.org](https://standardebooks.org/) .
495+
The volunteer-driven Standard Ebooks project relies on readers like you to submit typos, corrections, and other improvements. Anyone can contribute at [standardebooks.org](https://standardebooks.org/).
496496

497497
## Uncopyright
498498

@@ -504,4 +504,4 @@ Copyright pages exist to tell you that you *can't* do something. Unlike them, th
504504

505505
Copyright laws are different all over the world, and the source text or artwork in this ebook may still be copyrighted in other countries. If you're not located in the United States, you must check your local laws before using this ebook. Standard Ebooks makes no representations regarding the copyright status of the source text or artwork in this ebook in any country other than the United States.
506506

507-
Non-authorship activities performed on items that are in the public domain-so-called "sweat of the brow" work-don't create a new copyright. That means that nobody can claim a new copyright on an item that is in the public domain for, among other things, work like digitization, markup, or typography. Regardless, the contributors to this ebook release their contributions under the terms in the [CC0 1.0 Universal Public Domain Dedication](https://creativecommons.org/publicdomain/zero/1.0/) , thus dedicating to the worldwide public domain all of the work they've done on this ebook, including but not limited to metadata, the titlepage, imprint, colophon, this Uncopyright, and any changes or enhancements to, or markup on, the original text and artwork. This dedication doesn't change the copyright status of the source text or artwork. We make this dedication in the interest of enriching our global cultural heritage, to promote free and libre culture around the world, and to give back to the unrestricted culture that has given all of us so much.
507+
Non-authorship activities performed on items that are in the public domain-so-called "sweat of the brow" work-don't create a new copyright. That means that nobody can claim a new copyright on an item that is in the public domain for, among other things, work like digitization, markup, or typography. Regardless, the contributors to this ebook release their contributions under the terms in the [CC0 1.0 Universal Public Domain Dedication](https://creativecommons.org/publicdomain/zero/1.0/), thus dedicating to the worldwide public domain all of the work they've done on this ebook, including but not limited to metadata, the titlepage, imprint, colophon, this Uncopyright, and any changes or enhancements to, or markup on, the original text and artwork. This dedication doesn't change the copyright status of the source text or artwork. We make this dedication in the interest of enriching our global cultural heritage, to promote free and libre culture around the world, and to give back to the unrestricted culture that has given all of us so much.
Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
# HTML Text Formatting Examples
22

3-
This is a **bold (b)** example and right next to it we have a **strong emphasis (strong)** . Notice that **strong + bold mixed** looks similar but carries additional semantic meaning.
3+
This is a **bold (b)** example and right next to it we have a **strong emphasis (strong)**. Notice that **strong + bold mixed** looks similar but carries additional semantic meaning.
44

5-
Here is an *italic (i)* word and an *emphasis (em)* example. Sometimes we combine them like *italic + emphasis together* .
5+
Here is an *italic (i)* word and an *emphasis (em)* example. Sometimes we combine them like *italic + emphasis together*.
66

7-
Now let's look at text that appears crossed out: ~~strikethrough with s~~ and ~~deleted with del~~ . You can also mix them: ~~double strikethrough (s + del)~~ .
7+
Now let's look at text that appears crossed out: ~~strikethrough with s~~ and ~~deleted with del~~. You can also mix them: ~~double strikethrough (s + del)~~.
88

9-
To highlight insertions or underlines: underlined with u , inserted with ins . A combination could be: underline + insertion together .
9+
To highlight insertions or underlines: underlined with u, inserted with ins. A combination could be: underline + insertion together.
1010

11-
Subscript and superscript examples: Water is written as H 2 O using sub. The mathematical expression x 2 + y 3 uses sup. They can also be combined: CO 2 * .
11+
Subscript and superscript examples: Water is written as H 2 O using sub. The mathematical expression x 2 + y 3 uses sup. They can also be combined: CO 2 *.
1212

13-
Mixing several: This sentence has ***strong + emphasis*** , some **bold + underline** , and a formula like a 2 + b 3 .
13+
Mixing several: This sentence has ***strong + emphasis***, some **bold + underline**, and a formula like a 2 + b 3.

crates/fleischwolf/tests/data/html/expected/html_code_snippets.html.strict.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
# Code snippets
22

3-
The Pythagorean theorem can be written as an equation relating the lengths of the sides *a* , *b* and the hypotenuse *c* .
3+
The Pythagorean theorem can be written as an equation relating the lengths of the sides *a*, *b* and the hypotenuse *c*.
44

55
To use Docling, simply install `docling` from your package manager, e.g. pip: `pip install docling`
66

7-
To convert individual documents with python, use `convert()` , for example:
7+
To convert individual documents with python, use `convert()`, for example:
88

99
```
1010
from docling.document_converter import DocumentConverter
@@ -20,5 +20,5 @@ The program will output: `## Docling Technical Report[...]`
2020
Prefetch the models:
2121

2222
- Use the `docling-tools models download` utility:
23-
- Alternatively, models can be programmatically downloaded using `docling.utils.model_downloader.download_models()` .
23+
- Alternatively, models can be programmatically downloaded using `docling.utils.model_downloader.download_models()`.
2424
- Also, you can use download-hf-repo parameter to download arbitrary models from HuggingFace by specifying repo id: `$ docling-tools models download-hf-repo ds4sd/SmolDocling-256M-preview Downloading ds4sd/SmolDocling-256M-preview model from HuggingFace...`
Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
# Something
22

3-
Please follow the link to: [This page](#) .
3+
Please follow the link to: [This page](#).
Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Anchor Links Test
22

3-
Jump to [Section 2](#section-2) or visit [Example](https://example.com/) .
3+
Jump to [Section 2](#section-2) or visit [Example](https://example.com/).
44

55
## Section 2
66

7-
Content for section 2 with a [top link](#) .
7+
Content for section 2 with a [top link](#).

0 commit comments

Comments
 (0)