W108 purpose (warning about no POS annotation)

[tc64]

Is the following warning meant to be triggered only upon inferring the user wants to access the pos attribute? Or is the intent for this warning to be shown in all cases where there is no tagger or other lemma assigner in the pipeline?

[W108] The rule-based lemmatizer did not find POS annotation for the token '.'. Check that your pipeline includes components that assign token.pos, typically 'tagger'+'attribute_ruler' or 'morphologizer'.

Your Environment

• spaCy version: 3.0.0rc5
• Platform: Linux-4.19.0-13-cloud-amd64-x86_64-with-glibc2.10
• Python version: 3.8.5
• Pipelines: en_core_web_sm (3.0.0a1)

[adrianeboyd]

This warning means that you have a lemmatizer that won't work in your pipeline, so you either want to re-enable the components that provide pos or you want to remove the lemmatizer. The lemmatizer will still run, but it won't provide useful lemmas (just lowercased token texts).

In the en pipelines, you need to enable both the tagger and the attribute_ruler components to provide pos for the lemmatizer. In v2 the lemmas used to be assigned by the tagger, so this kind of inter-dependency between the pipeline components in v3 is new and potentially confusing, so we added a warning when we know the lemmatizer isn't going to work.

Or did you run into this error with the default en_core_web_sm pipeline? That would be unexpected and potentially a bug...

[tc64]

@adrianeboyd thanks for the info, I didn't realize lemmatizer was it's own pipe, I guess this is new in 3.0... Will try to disable that one as well (had been disabling tagger and attribute_ruler