How to get a list of all ent_type and pos integers for a model

[alexandrnikitin]

I want to get a list of all possible values ent_type and pos can take and dictionaries from int to str and str to int for a model, e.g. en_core_web_lg. How can I do it?

[svlandeg]

Hi!

I want to get a list of all possible values ent_type and pos can take

Builtin components in spaCy typically support the labels property, so you can do:

    nlp = spacy.load("en_core_web_lg")
    print(f"NER labels: {nlp.get_pipe('ner').labels}")
    print(f"Tagger labels: {nlp.get_pipe('tagger').labels}")

Which will give you

NER labels: ('CARDINAL', 'DATE', 'EVENT', 'FAC', 'GPE', 'LANGUAGE', 'LAW', 'LOC', 'MONEY', 'NORP', 'ORDINAL', 'ORG', 'PERCENT', 'PERSON', 'PRODUCT', 'QUANTITY', 'TIME', 'WORK_OF_ART')

Tagger labels: ('$', "''", ',', '-LRB-', '-RRB-', '.', ':', 'ADD', 'AFX', 'CC', 'CD', 'DT', 'EX', 'FW', 'HYPH', 'IN', 'JJ', 'JJR', 'JJS', 'LS', 'MD', 'NFP', 'NN', 'NNP', 'NNPS', 'NNS', 'PDT', 'POS', 'PRP', 'PRP$', 'RB', 'RBR', 'RBS', 'RP', 'SYM', 'TO', 'UH', 'VB', 'VBD', 'VBG', 'VBN', 'VBP', 'VBZ', 'WDT', 'WP', 'WP$', 'WRB', 'XX', '_SP', '``')

The latter list are the tag_ attributes of the tokens. The component creating the pos_ annotations, is the attribute_ruler, and you can inspect exactly how it does so like this:

    for pattern in nlp.get_pipe('attribute_ruler').patterns:
        for key_p, value_p in pattern.items():
            print(f"{key_p} --> {value_p}")

Which gives you output like

patterns --> [[{'TAG': '.'}]]
attrs --> {'POS': 'PUNCT', 'MORPH': 'PunctType=Peri'}
index --> 0

Basically telling you that a . tag would get parsed into a PUNCT POS.

and dictionaries from int to str and str to int for a model, e.g. en_core_web_lg. How can I do it?

These conversions are done by the StringStore:

stringstore = nlp.vocab.strings

You can query it directly:

print(stringstore["apple"])
print(stringstore[8566208034543834098])

or loop through all its items:

    for string in stringstore:
        print(f"string {string} -> {stringstore[string]}")