What do the labels (["I", "S"]) mean in SentenceRecognizer senter?

[Phat-Loc]

For the SentenceRecognizer, what does "I" stand for? "S" I assume is the start of sentence. I am planning to use Prodigy Nightly to fine tune the SentenceRecognizer.

What is reasoning behind having 3 different ways of specifying sent_start?

1. labels ["I", "S"]
2. sent_start flag
3. SENT_START annotation

[adrianeboyd]

I've written docs for the new sent recipes, but I'm not sure they've made it anywhere public for users yet.

S means sentence start and I means sentence-internal, like the I in IOB.

The senter model is a word-level tagger just like the tagger, it just happens to learn Token.is_sent_start as I/S instead of Token.tag like NN or JJ. When the annotations are set, it just sets Token.is_sent_start as False/True for I/S instead of setting a tag. For spacy it's really an internal detail that you don't need to know about (you never see it when you train or run the senter), but prodigy is currently using the same annotation interface as for the tagger recipes (pos and pos_manual), so it ends up displaying the internal tags.

We've thought a bit about how we could customize further for senter (suggestions are welcome!), for instance only showing the S tags, but for now it's still using the unmodified pos interface. When I'm using it, I set up custom colors so I is very light gray and S is a color that stands out like orange to make it easier to distinguish them.

[Phat-Loc]

Thanks! I am leaning towards only showing the S tag. I might write a UI that lets me set the sentence start any where in the document. Most of the documents that I use senter on have lists and sub lists e.g. 1. 2. (a.) (b.) etc... Also have a bunch with multiple \n characters spaced together. Any suggestions on handling multiple \n characters? Should I pre-process and remove them all or just leave replace multiple consecutive \n with just one.