German - Characters added to result multiple times (aä / AÄ)

[TheSeiko]

tesseract 4.00.00alpha
leptonica-1.74.1
libgif 4.1.6(?) : libjpeg 8d (libjpeg-turbo 1.5.0) : libpng 1.6.20 : libtiff 4.0.6 : zlib 1.2.8 : libwebp 0.4.3 : libopenjp2 2.1.0
Win10 64bit - built Uni Mannheim

deu.traineddata - Repeating of characters:

Current Behavior:

ÄGYPTEN -> ÄAGYPTEN
Grand-Prix -> Gräand-Prix
AUSTRALIEN -> AUSTRAÄLIEN
GROSSBRITANNIEN -> GROSSBRITAÄANNIEN

Expected Behavior:

ÄGYPTEN -> ÄGYPTEN
Grand-Prix -> Grand-Prix
AUSTRALIEN -> AUSTRALIEN
GROSSBRITANNIEN -> GROSSBRITANNIEN

Suggested Fix:
1 blob / 1 box should only be 1 outcome / 1 result

Additional Info:
Example images are available for posting

[TheSeiko]

Additional example (wW)
VW-Werk -> VwW-Werk

[Shreeshrii]

Please test with the latest models available in https://github.com/tesseract-ocr/tessdata/tree/master/best ShreeDevi

…

____________________________________________________________ भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

On Tue, Aug 1, 2017 at 9:45 PM, TheSeiko ***@***.***> wrote: Additional example (wW) VW-Werk -> VwW-Werk — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1060 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE2_o9N-1P4rP3fa2mYMcL4zS0Z8LMEYks5sT08WgaJpZM4Op9eR> .

[amitdo]

Suggested Fix:
1 blob / 1 box should only be 1 outcome / 1 result

1. It won't work with ligatures.
2. With the legacy OCR engine, there is a character segmentation step, and the OCR is done on individual char blobs.
   With the new LSTM engine, the OCR is done by the neural network on sequence of pixels in text lines, not on pre-segmented blobs.

[amitdo]

The fix for most problems with the LSTM engine is more / better training.

DAS2016 Sildes, 6. Modernization Efforts Page 17
Encyclopedia -> EE-n-c-yy-c-l-o-p-e-d-i-a -> Encyclopedia

I think that for 'in dictionary' words these kind of duplications would be eliminated.

[amitdo]

Similar issues: #884 #1011

[TheSeiko]

@Shreeshrii

https://github.com/tesseract-ocr/tessdata/tree/master/best
is not working @ALL

deu.traineddata 19.721 KB
best - deu.traineddata 8.427 KB

best trainingdata only delivers empty results

[Shreeshrii]

Are you using --oem 1?

you can see the contents of the traineddata by

combine_tessdata -u deu,traineddata

These are probably only lstm models and do not have the legacy engine which is used via --oem 0

[Shreeshrii]

@stweil Have you tested the deu model?

[TheSeiko]

Yes, I'm using --oem 1

I'm just switching deu.traineddata in tessdata
Old one works without problems, new one -> empty output
Checked the download - downloaded File has same size as in the repository.

[Shreeshrii]

@stweil - you may need to update the windows binaries on Uni Mannheim site with the latest updates from Ray.

@TheSeiko I haven't personally tested the deu model. WIll check and post result. Wondering whether your Windows binary is old....

[Shreeshrii]

Looks like you need both deu and frk models

wget -O ./tess4data-save/deubest.traineddata https://github.com/tesseract-ocr/tessdata/blob/master/best/deu.traineddata?raw=true

sudo cp ./tess4data-save/*.traineddata /usr/share/tesseract-ocr/4.00/tessdata

time tesseract ./tif/phototest.tif stdout --oem 1 -l deu
time tesseract ./tif/phototest.tif stdout --oem 1 -l deubest

Page 1
This is a lot of 12 point text to test the
ocr code and see if it works on all types
of file format.
The quick brown dog jumped over the
lazy fox. The quick brown dog jumped
over the lazy fox. The quick brown dog
jumped over the lazy fox. The quick
brown dog jumped over the lazy fox.
real	0m1.633s
user	0m2.032s
sys	0m0.492s
Error opening data file /usr/share/tesseract-ocr/4.00/frk.traineddata
Please make sure the TESSDATA_PREFIX environment variable is set to your "tessdata" directory.
Failed loading language 'frk'
Page 1
This is a lot of 12 point text to test the
ocr code and see if it works on all types
of file format.
The quick brown dog jumped over the
lazy fox. The quick brown dog jumped
over the lazy fox. The quick brown dog
Jumped over the lazy fox. The quick
brown dog jumped over the lazy fox.
real	0m2.045s
user	0m2.744s
sys	0m0.612s

[Shreeshrii]

works on linux - looks for frk traineddata, probably listed in deu.config

[stweil]

Have you tested the deu model?

The new best one? No, I have not tested it yet. I am currently focused on Fraktur where the new results clearly beat the old ones.

[...] update the windows binaries on Uni Mannheim site [...]

I noticed on Linux that "old" Tesseract executables crash with the new traineddata, so I expect that my current Windows binaries would crash, too. Building new ones is on my list.

[TheSeiko]

thank you

[stweil]

[...] update the windows binaries on Uni Mannheim site [...]

The new binaries are now available.
I now use semantic versioning, so this is my 4.0.0-alpha.20170804.