Annotation mistakes in STN?

[Hotown]

Hello, I found some annotation mistakes in SynthTabNet, e.g., in /marketing/images/train/img_000000_1634629424_098128.png, the cell with index 14 is annotated as an non-empty cell, but it is empty cell in the corresponding image.

Could you please tell me what should I do if I want to using SynthTabNet in other works? Are there any tricks in STN data preparing?

[nikos-livathinos]

Thank you for pointing out this issue, this is a bug in the annotations.

We have released the version v2.0.0 of SynthTabNet with corrected annotations (the images are the same as v1.0.0).

You can download it from: https://github.com/IBM/SynthTabNet#download

[Hotown]

Thanks for your reply! We will try new version later.

[aishniparab]

The new version has malformed strings and won't load correctly. Is anyone else facing this issue?

[nikos-livathinos]

@aishniparab If you could point out which part of SynthTabNet is failing to load, it would be greatly appreciated.
Additionally, providing some examples might help us to identify the issue.

[aishniparab]

I used the following code to read data from jsonl:

def load_jsonl(filepath):
    if not os.path.isfile(filepath):
        raise FileNotFoundError(f'Invalid filepath {filepath}')
    with open(filepath, "r", encoding="utf-8") as f:
        samples = f.readlines()
        valid_jsons = []
        errors = []
        for i, sample in enumerate(samples):
            try:
                json_obj = json.loads(sample)
                valid_jsons.append(json_obj)
            except Exception as e:
                errors.append((i, e))
        return valid_jsons, errors

All 150,000 samples from sparse_synthetic_data.jsonl were loaded.
All 150,000 samples from pubtabnet_synthetic_data.jsonl were loaded.

Only 147,946 samples from marketing_synthetic_data.jsonl were loaded, 2054 produced errors.
marketing_jsonl_read_errors.csv

Only 145,460 samples from fintabnet_synthetic_data.jsonl were loaded, 4540 produced errors.
fintabnet_jsonl_read_errors.csv