using data debug: how to get which document is corrupted?

[DSLituiev]

I've ran into an issue after running a model for 1200 samples:

ValueError: [E093] token.ent_iob values make invalid sequence: I without B

Now I'm trying to figure out which document caused the havoc.
I'm running python -m spacy debug data config-filled.cfg --code-path functions.py -V
but what I get is again same error without explanation of which document is corrupted.

How can I make it to show where the problem exactly is?

I used code with doc.set_ents:

from spacy.tokens import Doc, DocBin
doc_w_ents = defaultdict(lambda: dict())
docbin = DocBin()
for id, doc in ... :
        doc.set_ents([doc.char_span(ent[0], ent[1], label=ent[2], vector=None) for ent in  entities])
        docbin.add(doc)
       doc_w_ents[id]  = doc

I tried to check it manually:

for acc, doc in doc_w_ents.items():
    for span in doc.ents:
        if not span[0].ent_iob_ == 'B':
            raise ValueError

and it looks good, no "I" w/o "B".
How can I proceed here?

[polm]

I'm having trouble finding it, but we recently had a similar issue caused by having max_length not equal to 0 in the configs. If you're unlucky the splitting can split entities and cause issues. We recently changed the default max_length to fix that, but if your config isn't brand new that might be an issue.

Adding some output to help identify the document at issue here would be good, I'll have a look at that.

[polm]

Ah, this is the issue I was thinking of. #8026 (comment)

[DSLituiev]

Thank you. I realized in my case I have not updated the .spacy data files after a data fix (which probably had an issue that you're referring to), so the dataset I checked in python is different from the one I ran spacy data debug on. Now it's running OK.

[polm]

Hey, one question - I was looking at making the output include the text of the problem to allow for easier debugging, and it seems like it should already include that (and it's not a recent change). On the line after the E093 there should have been a line like This|O is|O text|O stuff|I. Did you not see anything like that?

[DSLituiev]

I do see a string like This|O is|O text|O stuff|I when training fails

[DSLituiev]

What I observe now, is that

python -m spacy init fill-config config.cfg config-filled.cfg

runs smoothly, but

python -m spacy train config-filled.cfg  --code ./functions.py \
    --output checkpoints-general-onco-frac/

returns an error:

ValueError: [E093] token.ent_iob values make invalid sequence: I without B
pole|I and|I mid|I kidney|I

Any ideas how to debug from here?

[DSLituiev]

max_length seem to be the culprit