CJK text handling

[rupertj]

Some PDFs from West Lindsey fail to import, as they contain Chinese text, which smalot/pdfparser fails to decode properly.

Example: Risk Strategy 2025.pdf

The text is on the last page: " 欲了解更多信息，请致电". Decoding fails after the first two chars, leaving this: "欲了9�+�"p2���^�T�G�", which fails to insert into the database, with the error: "Incorrect string value: '\x83+\xA9".

This needs fixing in the pdfparser, but we could also handle the failure better - maybe by removing the broken text so the rest of the document saves (and logging that we did so).

[rupertj]

I've dug into this a bit more, using this PDF:

West Lindsey Cultural Strategy.pdf

With reference to this excellent StackOverflow answer about how text is encoded in PDFs, and this tool that lets you explore the raw data in a pdf easily, I've found the problematic text and decoded it manually.

Here's the encoded text:

BT
/C2_1 1 Tf
-0.266 Tc 0.266 Tw 15 0 0 15 296.7708 342.6935 Tm
[<3DC1>-168.9 <2115>-140.2 <5C72>-225.2 <3983>-247.8 <2BA9>-250.8 <2270>-220.1 <32FE>-58.4 <19E85E86>-234.7 <5483>-206.1 <47C4>]TJ
ET

(In the explorer: Trailer, Root, Pages, Kids, 4, Kids, 9, Contents, )

The sequence "/C2_1 1 Tf" means that this text should be decoded using Font C2_1. This font contains a ToUnicode entry that shows which chars in the encoding map to which unicode values. Here's the map:

/CIDInit /ProcSet findresource begin
12 dict begin
begincmap
/CIDSystemInfo
<< /Registry (Adobe)
/Ordering (UCS) /Supplement 0 >> def
/CMapName /Adobe-Identity-UCS def
/CMapType 2 def
1 begincodespacerange
<0000> <FFFF>
endcodespacerange
11 beginbfchar
<2115> <4E86>
<2270> <4FE1>
<5C72> <89E3>
<3983> <66F4>
<5483> <81F4>
<5E86> <8BF7>
<2BA9> <591A>
<3DC1> <6B32>
<47C4> <7535>
<19E8> <FF0C>
<32FE> <606F>
endbfchar
endcmap CMapName currentdict /CMap defineresource pop end end

(In the explorer: Trailer, Root, Pages, Kids, 4, Kids, 9, Resources, Font, C2_1, ToUnicode, )

If we take the first 3 chars of the text from the unencoded text:

3DC1 2115 5C72

Then look up their unicode mappings:

6B32 4E86 89E3

Then look up those unicode chars:

欲了解

This matches the first three chars of the text.

So, smalot/pdfparser isn't following this process properly. The next step is to figure out exactly what it's doing incorrectly.