feat(pdf): RTL reading order + Arabic shaping + gap-aware cell join

Improves right-to-left (Arabic) extraction and tightens table-PDF assembly:

- RTL reading order: cells on a line read right→left, so an Arabic-majority
  region now sorts each line band by descending left edge (was always LTR).
- RTL line wrapping: line splitting is now direction-aware — RTL wraps reset x
  rightward (`g.l > p.r`), so wrapped Arabic paragraphs no longer merge into one
  overlapping cell.
- RTL word gaps: split at the right-to-left inter-word gap (`p.l - g.r`) rather
  than trusting pdfium's space glyphs, which it spuriously inserts inside Arabic
  words.
- Arabic lam-alef: pdfium decomposes the إ/أ/آ-lam ligatures in visual order
  (`alef, lam`); swap mid-word back to logical `lam, alef`. Restricted to the
  hamza/madda variants — plain `alef+lam` is ambiguous (article vs ligature).
- Arabic↔Latin boundary: insert the space pdfium runs together (`وPython` →
  `و Python`).
- Gap-aware cell join (all scripts): same-band cells join without a space when
  abutting (so a word split across segments isn't broken), with a space across a
  real gap or line break.

right_to_left_01 4→2, right_to_left_03 76→70, and the table papers tighten
(2206 210→200, redp5110 300→298, 2203 313→309). The 3 exact PDFs are unchanged.
No-op for non-Arabic text. Regenerated affected snapshot fixtures.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: artiz

crates/fleischwolf-pdf/examples/dump_regions.rs

@@ -40,9 +40,15 @@ fn main() {
                 tail
             );
         }
-        // cells near the page bottom, to spot orphans below the last region
-        for c in page.cells.iter().filter(|c| c.t > 595.0 && c.t < 660.0) {
-            println!("   CELL t={:6.1} b={:6.1} | {}", c.t, c.b, c.text);
+        // raw line cells in extraction order (to inspect RTL ordering)
+        if std::env::var("DUMP_CELLS").is_ok() {
+            for (ci, c) in page.cells.iter().enumerate() {
+                let snip: String = c.text.chars().take(50).collect();
+                println!(
+                    "   CELL[{ci}] t={:6.1} l={:6.1} r={:6.1} | {}",
+                    c.t, c.l, c.r, snip
+                );
+            }
         }
     }
 }

crates/fleischwolf-pdf/src/assemble.rs

@@ -101,7 +101,8 @@ fn order_regions<T>(items: &mut [T], page_w: f32, reg: impl Fn(&T) -> &Region) {
 /// `{ ahn }`, `Name 1 .`, `[ 9 ]`), and a geometric gap heuristic diverges from
 /// it more than a plain single-space join does.
 fn clean_text(text: &str) -> String {
-    text.replace("\u{2} ", "")
+    let out = text
+        .replace("\u{2} ", "")
         .replace("\u{ad} ", "")
         .replace(['\u{2}', '\u{ad}'], "") // any stray wrap hyphens not at a join
         .replace(['\u{2018}', '\u{2019}'], "'") // ‘ ’ → '
@@ -110,7 +111,60 @@ fn clean_text(text: &str) -> String {
         .replace('\u{2026}', "...") // … → ...
         .split_whitespace()
         .collect::<Vec<_>>()
-        .join(" ")
+        .join(" ");
+    fix_arabic_lam_alef(&out)
+}
+
+/// pdfium decomposes the Arabic lam-alef ligature (لا / لإ / لأ / لآ) into its
+/// glyph constituents in *visual* order — `alef-variant, lam` — but docling keeps
+/// logical order, `lam, alef-variant`. Swap a mid-word `alef-variant + lam` back
+/// to `lam + alef-variant`. "Mid-word" (the previous char is an Arabic letter)
+/// distinguishes the ligature from the definite article `ال` (word-initial
+/// `alef + lam`), which must stay. No-op for non-Arabic text.
+fn fix_arabic_lam_alef(s: &str) -> String {
+    let is_arabic_letter = |c: char| ('\u{0620}'..='\u{064A}').contains(&c);
+    let chars: Vec<char> = s.chars().collect();
+    if !chars.iter().any(|&c| is_arabic_letter(c)) {
+        return s.to_string(); // no-op for non-Arabic text
+    }
+    // Pass 1: swap mid-word `alef-variant + lam` → `lam + alef-variant`. Only the
+    // hamza/madda alef variants (إ أ آ) are safe: the definite article is always
+    // plain `ا + ل`, so plain `alef + lam` is ambiguous (a legitimate `فعالة` vs a
+    // reversed `لا` ligature look identical) — leaving plain alef alone avoids
+    // corrupting legitimate words.
+    let mut a: Vec<char> = Vec::with_capacity(chars.len());
+    let mut i = 0;
+    while i < chars.len() {
+        let c = chars[i];
+        if matches!(c, '\u{0622}' | '\u{0623}' | '\u{0625}')
+            && chars.get(i + 1) == Some(&'\u{0644}')
+            && i > 0
+            && is_arabic_letter(chars[i - 1])
+        {
+            a.push('\u{0644}');
+            a.push(c);
+            i += 2;
+            continue;
+        }
+        a.push(c);
+        i += 1;
+    }
+    // Pass 2: insert a space at Arabic↔Latin boundaries (bidi script switch) that
+    // pdfium runs together — docling separates the embedded Latin run (`وPython`
+    // → `و Python`).
+    let mut out: Vec<char> = Vec::with_capacity(a.len());
+    for (j, &c) in a.iter().enumerate() {
+        if j > 0 {
+            let p = a[j - 1];
+            if (is_arabic_letter(p) && c.is_ascii_alphabetic())
+                || (p.is_ascii_alphabetic() && is_arabic_letter(c))
+            {
+                out.push(' ');
+            }
+        }
+        out.push(c);
+    }
+    out.into_iter().collect()
 }
 
 /// Cells assigned to a region (best container), in reading order, joined.
@@ -123,19 +177,49 @@ fn region_text(region: &Region, cells: &[TextCell]) -> String {
         })
         .collect();
     // Quantize the top coordinate into ~line bands so cells on the same line
-    // sort left-to-right; this is a strict total order (a raw fuzzy comparator
-    // is not transitive and makes Rust's sort panic).
+    // sort in reading order; this is a strict total order (a raw fuzzy comparator
+    // is not transitive and makes Rust's sort panic). For a right-to-left
+    // (Arabic-majority) region, cells on a line read right→left, so sort the band
+    // by descending left edge.
     let band = inside
         .iter()
         .map(|c| (c.b - c.t).abs())
         .fold(0.0f32, f32::max)
         .max(1.0);
-    inside.sort_by_key(|c| ((c.t / band).round() as i64, (c.l * 10.0) as i64));
-    let joined = inside
+    let arabic = inside
         .iter()
-        .map(|c| c.text.trim())
-        .collect::<Vec<_>>()
-        .join(" ");
+        .flat_map(|c| c.text.chars())
+        .filter(|&c| ('\u{0600}'..='\u{06FF}').contains(&c))
+        .count();
+    let latin = inside
+        .iter()
+        .flat_map(|c| c.text.chars())
+        .filter(|c| c.is_ascii_alphabetic())
+        .count();
+    let rtl = arabic > latin;
+    inside.sort_by_key(|c| {
+        let x = (c.l * 10.0) as i64;
+        ((c.t / band).round() as i64, if rtl { -x } else { x })
+    });
+    // Join cells in reading order. Cells on different line-bands join with a
+    // space (line break). Cells on the same band join with a space only if there
+    // is a real horizontal gap between them — an RTL line is split into adjacent
+    // segments mid-word (`الت`|`ي` → `التي`), so abutting same-band cells must not
+    // get a spurious space.
+    let mut joined = String::new();
+    let mut prev: Option<&&TextCell> = None;
+    for c in &inside {
+        if let Some(p) = prev {
+            let same_band = ((p.t / band).round() as i64) == ((c.t / band).round() as i64);
+            let h = (c.b - c.t).abs().max((p.b - p.t).abs()).max(1.0);
+            let gap = if rtl { p.l - c.r } else { c.l - p.r };
+            if !same_band || gap > h * 0.25 {
+                joined.push(' ');
+            }
+        }
+        joined.push_str(c.text.trim());
+        prev = Some(c);
+    }
     clean_text(&joined)
 }
 

crates/fleischwolf-pdf/src/pdfium_backend.rs

@@ -318,7 +318,15 @@ fn lines_from_glyphs(gs: &[Glyph], page_h: f32, code: bool) -> Vec<TextCell> {
             // *large* drop (≥1.5× the line height — a skipped line, e.g. a centered
             // page-number footer below a short last word) is always a new line,
             // even without the x-reset.
-            new_line = (p.b - g.b > h * 0.5 && g.l < p.r) || (p.b - g.b > line_h.max(h) * 1.5);
+            // LTR wraps reset x leftward (`g.l < p.r`); RTL (Arabic) wraps reset
+            // rightward (the new line begins at the far right). A large drop
+            // (≥1.5× line height) is a new line regardless of x.
+            let x_reset = if is_arabic(g.ch) || is_arabic(p.ch) {
+                g.l > p.r
+            } else {
+                g.l < p.r
+            };
+            new_line = (p.b - g.b > h * 0.5 && x_reset) || (p.b - g.b > line_h.max(h) * 1.5);
             // Don't split before closing punctuation, after opening punctuation, or
             // after a period that runs into a digit/lowercase letter — docling
             // keeps `engines,` / `[37` / `i.e.` / `98.5` together even across a
@@ -332,6 +340,12 @@ fn lines_from_glyphs(gs: &[Glyph], page_h: f32, code: bool) -> Vec<TextCell> {
             let word_gap = line_h.max(h) * 0.25;
             new_word = if code {
                 new_line || pending_space
+            } else if is_arabic(g.ch) || is_arabic(p.ch) {
+                // RTL runs right-to-left, so the inter-word gap is `p.l - g.r`. A
+                // real word space has a gap; pdfium also emits spurious zero-gap
+                // space glyphs inside words (`التي`), so require the gap rather
+                // than trusting a bare space glyph.
+                new_line || (p.l - g.r > word_gap && !glued)
             } else {
                 new_line || ((pending_space || g.l - p.r > word_gap) && !glued)
             };
@@ -389,7 +403,15 @@ fn words_from_glyphs(gs: &[Glyph], page_h: f32) -> Vec<TextCell> {
         let mut new_line = false;
         let mut new_word = false;
         if let Some(p) = prev {
-            new_line = (p.b - g.b > h * 0.5 && g.l < p.r) || (p.b - g.b > line_h.max(h) * 1.5);
+            // LTR wraps reset x leftward (`g.l < p.r`); RTL (Arabic) wraps reset
+            // rightward (the new line begins at the far right). A large drop
+            // (≥1.5× line height) is a new line regardless of x.
+            let x_reset = if is_arabic(g.ch) || is_arabic(p.ch) {
+                g.l > p.r
+            } else {
+                g.l < p.r
+            };
+            new_line = (p.b - g.b > h * 0.5 && x_reset) || (p.b - g.b > line_h.max(h) * 1.5);
             // No digit-digit glue here (unlike the prose grouping): table cells in
             // adjacent columns are numeric and a column gap must still split them
             // (`0.965` `0.934`, not `0.9650.934`). Intra-number digits have no gap
@@ -436,6 +458,10 @@ fn words_from_glyphs(gs: &[Glyph], page_h: f32) -> Vec<TextCell> {
     cells
 }
 
+fn is_arabic(c: char) -> bool {
+    ('\u{0600}'..='\u{06FF}').contains(&c)
+}
+
 fn is_close_punct(c: char) -> bool {
     matches!(
         c,

tests/pdf_snapshots/latex/sources/2305.03393/llncsdoc.pdf.md

@@ -159,7 +159,7 @@ From a technical point of view, the llncs document class does not require any sp
 The llncs document class supports some additional special characters:
 
 ```
-\grole yields > < \getsto yields ← → \lid yields < = \gid yields > =
+\grole yields >< \getsto yields ← → \lid yields <= \gid yields >=
 ```
 
 If you need blackboard bold characters, i.e. for sets of numbers, please load the related AMS-TEXfonts. If for some reason this is not possible you can also use the following commands from the llncs class:
@@ -192,7 +192,7 @@ cas e (env.) Other theorem-like environments render the text in roman, while the
 example (env.)
 
 ```
-property(env.) question(env.) exercise(env.) solution(env.)heading is bold as well: problem(env.) note(env.) \begin{case} <text> \end{case} \begin{conjecture} <text> \end{conjecture} \begin{example} <text> \end{example} \begin{exercise} <text> \end{exercise} \begin{note} <text> \end{note} \begin{problem} <text> \end{problem} remark(env.) \begin{property} <text> \end{property} \begin{question} <text> \end{question} \begin{remark} <text> \end{remark} \begin{solution} <text> \end{solution}
+property(env.)question(env.)exercise(env.)solution(env.)heading is bold as well:problem(env.)note(env.) \begin{case} <text> \end{case}\begin{conjecture} <text> \end{conjecture}\begin{example} <text> \end{example}\begin{exercise} <text> \end{exercise}\begin{note} <text> \end{note}\begin{problem} <text> \end{problem} remark(env.) \begin{property} <text> \end{property}\begin{question} <text> \end{question}\begin{remark} <text> \end{remark}\begin{solution} <text> \end{solution}
 ```
 
 claim (env.) Finally, there are also two unnumbered environments that have the run-in headproof (env.) ing in italics and the text in upright roman.

tests/pdf_snapshots/latex/sources/2412.19437/figures/relative_expert_load_multi_7-12.pdf.md

@@ -1,3 +1,3 @@
 <!-- image -->
 
-DM Mathematics Aux-Loss-Based Layer 12 12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364Wikipedia (en) Github
+DM Mathematics Aux-Loss-Based Layer 1212345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364Wikipedia (en)Github

tests/pdf_snapshots/mets_gbs/sources/32044009881525_select.tar.gz.md

@@ -1,16 +1,16 @@
-recently become prevalent that he who speaks of military power is a " militarist . " This , how- reverse assertion ever , is as great a fallacy as the that he who talks of nothing but peace is a 66 pacifist . "
+recently become prevalent that he who speaks of military power is a "militarist." This, how- reverse assertion ever, is as great a fallacy as the that he who talks of nothing but peace is a 66 pacifist."
 
-Truth , even bitter truth , is better than the most high - minded fallacy .
+Truth, even bitter truth, is better than the most high-minded fallacy.
 
-The author has visited Japan , Siberia , China , the Philippines , the Malay States , and Hawaii in 1919 and 1920 , and his personal impressions and investigations form the basis of the present book . The list of works which he has perused in the course of his investigation of the problem of the Pacific is hereto appended .
+The author has visited Japan, Siberia, China, the Philippines, the Malay States, and Hawaii in 1919 and 1920, and his personal impressions and investigations form the basis of the present book. The list of works which he has perused in the course of his investigation of the problem of the Pacific is hereto appended.
 
-The author wishes to acknowledge his debt to Admiral A. D. Bubnov , who has contributed Chapters VII - X . Admiral Bubnov took part in the Russo - Japanese War , was Professor of the Naval Staff College at Petrograd , and Chief of the Naval Section of the Staff of the Supreme Commander - in - Chief in the Great War . The Admiral is an authoritative student of the questions of naval strategy discussed in the chapters that belong to his pen .
+The author wishes to acknowledge his debt to Admiral A. D. Bubnov, who has contributed Chapters VII- X. Admiral Bubnov took part in the Russo-Japanese War, was Professor ofthe Naval Staff College at Petrograd, and Chief of the Naval Section of the Staff of the Supreme Commander-in-Chief in the Great War. The Admiral is an authoritative student of the questions of naval strategy discussed in the chapters that belong to his pen.
 
-The author also has to thank Mr. C. Nabokoff , the late Russian Chargé d'Affaires in London , book . for undertaking the translation of his
+The author also has to thank Mr. C. Nabokoff, the late Russian Chargé d'Affaires in London, book. for undertaking the translation of his
 
 ## 62 THE PROBLEM OF THE PACIFIC
 
-copper mines with up - to - date technical equip- ment in various provinces of Central China ¹ : -
+copper mines with up-to-date technical equip- ment in various provinces of Central China ¹:-
 
 |  |  |  |  |
 | - | - | - | - |
@@ -26,15 +26,15 @@ copper mines with up - to - date technical equip- ment in various provinces of C
 |  |  |  |  |
 |  |  |  |  |
 
-It has been mentioned in one of the preceding chapters that Japanese industries needed iron . The desire to obtain the necessary supplies of iron is thus perfectly legitimate . Japan's endea- vour to acquire concessions for the production of iron and coal is not , therefore , a proof of the Imperialism of her policy . But , as the French saying goes , Le ton fait la chanson . Japan is trying to get hold of the entire iron industry of China . She is doing so by the veiled seizure of political and administrative control of the respective provinces of China . To Europe and America this is presented under the guise of the nebulous formula of 66 special interests in the regions of China adjacent to Japan . " Man- churia and Shantung are first in that list . When Japan succeeds , she will have deprived China
+It has been mentioned in one of the preceding chapters that Japanese industries needed iron. The desire to obtain the necessary supplies of iron is thus perfectly legitimate. Japan's endea- vour to acquire concessions for the production of iron and coal is not, therefore, a proof of the Imperialism of her policy. But, as the French saying goes, Le ton fait la chanson. Japan is trying to get hold of the entire iron industry of China. She is doing so by the veiled seizure of political and administrative control of the respective provinces of China. To Europe and America this is presented under the guise of the nebulous formula of 66special interests in the regions of China adjacent to Japan." Man- churia and Shantung are first in that list. When Japan succeeds, she will have deprived China
 
-1 These data relate to 1915 .
+1 These data relate to 1915.
 
 ## 252 THE PROBLEM OF THE PACIFIC
 
-France may lay down new tonnage in the years 1927 , 1929 , and 1931 , as provided in Part 3 , Section II .
+France may lay down new tonnage in the years 1927, 1929, and 1931, as provided in Part 3, Section II.
 
-Ships which may be retained by Italy .
+Ships which may be retained by Italy.
 
 |  |  |  |  |
 | - | - | - | - |
@@ -50,7 +50,7 @@ Ships which may be retained by Italy .
 |  |  |  |  |
 |  |  |  |  |
 
-Italy may lay down new tonnage in the years 1927 , 1929 and 1931 , as provided in Part 3 , Section II .
+Italy may lay down new tonnage in the years 1927, 1929 and 1931, as provided in Part 3, Section II.
 
 |  |  |  |  |  |  |  |
 | - | - | - | - | - | - | - |
@@ -66,8 +66,8 @@ Italy may lay down new tonnage in the years 1927 , 1929 and 1931 , as provided i
 |  |  |  |  |  |  |  |
 |  |  |  |  |  |  |  |
 
-- Part 2. - RULES FOR SCRAPPING VESSELs of War . The following rules shall be observed for the scrapping of vessels of war which are to be disposed of in accordance with Articles II and III .
-- I. A vessel to be scrapped must be placed in such condition that it cannot be put to combative use .
-- II . This result must be finally effected in any one of the following ways :
-- ( a ) Permanent sinking of the vessel ;
-- ( b ) Breaking the vessel up . This shall always involve the destruction or removal of all machinery , boilers and armour , and all deck , side and bottom plating ;
+- Part 2.-RULES FOR SCRAPPING VESSELs of War. The following rules shall be observed for the scrapping of vessels of war which are to be disposed of in accordance with Articles II and III.
+- I. A vessel to be scrapped must be placed in such condition that it cannot be put to combative use.
+- II. This result must be finally effected in any one of the following ways:
+- (a) Permanent sinking of the vessel ;
+- (b) Breaking the vessel up. This shall always involve the destruction or removal of all machinery, boilers and armour, and all deck, side and bottom plating;

tests/pdf_snapshots/odf/sources/odf_presentation_01.odp.pdf.md

@@ -1,15 +1,15 @@
-## e e a e g r s s s ● e 1
+## e e a e gr ss s ● e 1
 
 ```
-eW e● a s e F l ● e T e● ● u e T r F e● ● ● s e F C● ● t C A 2● ● ● C ● a b e T l a s F l ● u r a s F l ● a s l a s e u e l T r ● ● B ● ● a s s l S d l i
+eWe●aseFl● eTe●● ueTrFe●●● seFC●● tCA2●●● C● abeTl asFl● ur asFl● asl aseuelTr●● B●● assl Sdli
 ```
 
 <!-- image -->
 
 - S
 
-A• :• S e t I m B e t I m o e o m i n f
+A•:•S etImB etIm oeominf
 
-- o t I t I m
+- o tI tI m
 - e e e m m
 - n f o

tests/pdf_snapshots/odf/sources/odf_presentation_02.odp.pdf.md

@@ -1,8 +1,8 @@
 <!-- image -->
 
-m d e b r a m s m i a u t a e i q l i u s , a p m m v c s v u t e a c e e u s a a f t v i v a l l u a , r t n r i u u e m u s c c n u d a u s c p l a r r . . o t c u e m p s b i u M M r m c o b i e u a t s e a a c s u l c r u m s e t m u r e o n a c t i . s . o . p s i c e d n u M s N d i e i n i m n s a o d u o . r d a e u n u a a l l V o l e s n s i r l e
+m d e br a m s m i a ut a e i qli us ,a p m m v c s v u te a c e e us a af tv iv a ll u a ,r tn ri u u e m us c c n u da us c pl arr . . ot cu e m p s bi u M M r m c o bi e u at se a a c s ul cr u m s et m ur e on a ct i. s . o . p si ce d n u M s N di e i n im n s a od u o. r d a e u n u a all V ol e s n si r l e
 
-m n m e a e t b a r . h n a i x p u e n t . o l M c i d m , o u l c N e r o u t i f v i t r m m u m r a l . I e a t t t i n b i v t n s l i e o e n o l c q i t i e , r q
+m n m e a e t b ar . h n ai x p u en t. ol M ci d m , o ul c N er o u ti f vi tr m m u m r al .I e at tti n bi v tn sli e o en ol c q i ti e , r q
 
 <!-- image -->