Best model to select for address recognition

[niels-przybilla]

Hello community,

we want to recognize and read addresses from german letters. We need only the receiver and the address of the receiver like
Company
Name
Street / number
Postcode City
Country

What is the best model to choose for such a usecase?

THX and best wishes

Niels

[felixdittrich92]

Hi @niels-przybilla 👋 ,

This is a KIE problem (doctr doesn't provides a pretrained KIE model yet) so i would suggest a two stage approach for a first MVP:

1. Use docTR for OCR to get the full text (line based: resolve_blocks=False -> .render())

• our best models actually are: fast_base for detection and parseq for recognition (to use the pretrained fast_base model you need to switch to the main branch (v0.9.0a0) -> pip3 install python-doctr[torch]@git+https://github.com/mindee/doctr.git
• all our recognition models are pretrained on french vocab so if you need "Umlaute" you could try: https://huggingface.co/Felix92/doctr-torch-parseq-multilingual-v1 which was fine tuned on a wider vocab including Umlate + ß

2. with the full text make use of GLiNER to extract the required entities:

• urchade/gliner_multi_pii-v1 works pretty well also for german

quick example:

import doctr
from doctr.io import DocumentFile
from doctr.models import ocr_predictor, from_hub
from gliner import GLiNER

print(doctr.__version__)
# --> v0.9.0a0

model = GLiNER.from_pretrained("urchade/gliner_multi_pii-v1")


doc = DocumentFile.from_images(["/home/felix/Desktop/doctr_test_data/003.jpg"])

predictor = ocr_predictor(
        det_arch="fast_base",
        reco_arch=from_hub("Felix92/doctr-torch-parseq-multilingual-v1"),  # <-- supports also Umlauts
        pretrained=True,
        det_bs=8,
        reco_bs=1024,
        assume_straight_pages=True,
        straighten_pages=False,
        detect_orientation=False,
        resolve_blocks=False,  # <-- set to False
        resolve_lines=True,
    ).cuda().half()  # <-- comment if you don't have a GPU
predictor.det_predictor.model.postprocessor.bin_thresh = 0.3
predictor.det_predictor.model.postprocessor.box_thresh = 0.1

result = predictor(doc)
full_text = result.render()


labels = ["Person", "Date"]
# Perform entity prediction

# Keep in mind that the model is limited to 300 tokens so for longer texts you may need to chunk the text + batch inference

entities = model.predict_entities(full_text, labels, threshold=0.5)

for entity in entities:
    print(entity["text"], "=>", entity["label"])

""" Console:
0.9.0a0

Mr. Anjum Hameed => Person
01-12-2006 => Date
"""