Understanding block segmentation after recognition

[ep0p]

I've encountered a curious case where a visually whole block is split in multiple ones in a way that seems illogical

In this example the recognition gives me 3 blocks :

1. for the the lines 1,3,4,5,6
2. for the 2nd line
3. for the 7th line

i was wondering how the segmentation is done ?

[felixdittrich92]

Hi @ep0p 👋

It takes the sorted boxes from the estimated lines and tries to build a cluster depending on the distance to estimate which lines could build a block.
ref.: https://github.com/mindee/doctr/blob/3f8bd0104411c671d9c9fa09934c06f970c7cb67/doctr/models/builder.py#L156C25-L156C30

For images like your provided one i suggest to disable the block resolving: resolve_blocks=False then it will only sort the boxes and lines :)

[ep0p]

Yes, the issue is that i was kind of hoping to be able to split the page into paragraphs. The layout is pretty complex and i need to be able to properly segment it.

Thank you for your answer.
Another question if i can post it here, if not let me know, i will open another thread: would it be possible to combine Doctr with other ocrs like Tesseract for example.
I'm asking because sometimes doctr can detect 2 words as one word or it does not recognize exactly.

[ep0p]

Thank you, @felixdittrich92, for your suggestions. The models 'fast_base' and 'parseq' are significant improvements over the ones I was using previously (db_resnet50, crnn_vgg16_bn). However, I am still experiencing the issue where two words are detected as one. I am using the main branch and my software is up-to-date.

As my document contains confidential information, I could send it to you via email if that’s acceptable.

[felixdittrich92]

Thank you, @felixdittrich92, for your suggestions. The models 'fast_base' and 'parseq' are significant improvements over the ones I was using previously (db_resnet50, crnn_vgg16_bn). However, I am still experiencing the issue where two words are detected as one. I am using the main branch and my software is up-to-date.

As my document contains confidential information, I could send it to you via email if that’s acceptable.

You can send it to felixdittrich92@gmail.com :)
You could give db_mobilenet_v3_large also a try it's the first model we have retrained from scratch with the new augmentation pipeline.

[ep0p]

Hi @felixdittrich92,

I have just sent the document and tested the db_mobilenet_v3_large model.

It performs significantly better than fast_base, accurately distinguishing some words that were previously merged together by fast_base. However, it does introduce new ones.

I believe implementing a LLM for OCR post-processing could effectively address these minor issues.

[felixdittrich92]

Hi @ep0p 👋🏼,

Yeah got it 👍🏼
Nice to hear about db_mobilenet_v3_large (because that's actually our smallest model), so we will also train the other models with the new aug pipeline in the next time step by step. (CC @odulcy-mindee )

2 suggestions:

• about the layout you should disable the blocks resolving resolve_blocks=True @odulcy-mindee This does not work well in the most cases (there is already an ticket [improvement] .render() isn't that robust - wrong ordered results #1586). Should we change the default to False ftm wdyt ?
• @ep0p You can play a bit around with predictor.det_predictor.model.postprocessor.bin_thresh = 0.5 # range 0.1 - 0.9 any float to further improve the results to your need.

Yeah i think you can easily fix the last small mistakes with an LM (or if you want to use the hammer xD an LLM) which fixes the "typos"

Best regards,
Felix

[ep0p]

Got it, @felixdittrich92 ! I'll disable blocks resolving as suggested.
Regarding the bin_thresh parameter, I've already experimented with it and noticed some improvements in the results. I'll continue tweaking it to fine-tune the performance.

I'm looking forward to the updates on the text render issue. Hopefully, it'll be resolved soon, which would make easy data feeding into the LLM :)

Thanks for your help!