Summary
I have observed inconsistent and linguistically unexpected lemma outputs for certain English verbs when using spaCy's en_core_web_sm and en_core_web_md models. In particular, some past tense or derived forms are mapped to incorrect lemmas that do not correspond to the expected base dictionary form.
This behavior affects downstream tasks that rely on lemma normalization, such as information retrieval and educational content generation.
🔍 Environment
spaCy version: 3.8.13
Model: en_core_web_sm (also tested with en_core_web_lg)
Python version: 3.14.6
OS: Windows-10-10.0.19045-SP0
❗ Observed Behavior
The following inconsistencies were observed:
Example 1
Input sentence:
The noise scared her.
Token analysis:
scared → lemma: scar ❌
Expected lemma: scare
Example 2 (additional observed words)
The issue is not limited to a single verb. Similar inconsistencies were observed with multiple verbs:
amaze
marry
web
...
In several cases, past tense or derived forms are mapped to lemmas that are not the expected base verb form.
📌 Expected Behavior
All inflected forms of a verb should ideally normalize to the same lemma:
scare → scare
scared → scare
scaring → scare
scares → scare
📌 Actual Behavior
Some inflected forms are mapped inconsistently:
scared → scar
This introduces ambiguity and breaks downstream normalization pipelines that rely on lemma consistency.
⚠️ Impact
This issue affects applications that depend on lemma normalization for:
Information retrieval systems
Language learning applications (e.g. flashcard generation)
Text indexing and search
Educational NLP pipelines
In such systems, inconsistent lemma outputs reduce reliability and require manual post-processing or custom normalization layers.
💡 Additional Notes
I understand that spaCy lemmatization is probabilistic and depends on lexical lookup tables and morphological rules. However, the observed behavior appears inconsistent for common English verbs and may benefit from:
improved lexical mapping consistency
or clarification of expected lemma behavior for derived verb forms
📎 Question
Is this behavior expected due to internal lexical design choices, or should these cases be considered lemmatization errors in the English model?
Additionally, is there a recommended best practice for handling cases where lemma output is inconsistent across inflected forms of the same verb?
If this behavior is expected and the issue is actually caused by my implementation rather than the English models, I would greatly appreciate it if someone could point that out.
I have made my project publicly available, and I would be grateful if you could take a quick look at the relevant code:
Repository:
Any guidance or suggestions would be greatly appreciated. Thank you for your time and for your work on spaCy.
Summary
I have observed inconsistent and linguistically unexpected lemma outputs for certain English verbs when using spaCy's en_core_web_sm and en_core_web_md models. In particular, some past tense or derived forms are mapped to incorrect lemmas that do not correspond to the expected base dictionary form.
This behavior affects downstream tasks that rely on lemma normalization, such as information retrieval and educational content generation.
🔍 Environment
spaCy version: 3.8.13
Model: en_core_web_sm (also tested with en_core_web_lg)
Python version: 3.14.6
OS: Windows-10-10.0.19045-SP0
❗ Observed Behavior
The following inconsistencies were observed:
Example 1
Input sentence:
The noise scared her.
Token analysis:
scared → lemma: scar ❌
Expected lemma: scare
Example 2 (additional observed words)
The issue is not limited to a single verb. Similar inconsistencies were observed with multiple verbs:
amaze
marry
web
...
In several cases, past tense or derived forms are mapped to lemmas that are not the expected base verb form.
📌 Expected Behavior
All inflected forms of a verb should ideally normalize to the same lemma:
scare → scare
scared → scare
scaring → scare
scares → scare
📌 Actual Behavior
Some inflected forms are mapped inconsistently:
scared → scar
This introduces ambiguity and breaks downstream normalization pipelines that rely on lemma consistency.
This issue affects applications that depend on lemma normalization for:
Information retrieval systems
Language learning applications (e.g. flashcard generation)
Text indexing and search
Educational NLP pipelines
In such systems, inconsistent lemma outputs reduce reliability and require manual post-processing or custom normalization layers.
💡 Additional Notes
I understand that spaCy lemmatization is probabilistic and depends on lexical lookup tables and morphological rules. However, the observed behavior appears inconsistent for common English verbs and may benefit from:
improved lexical mapping consistency
or clarification of expected lemma behavior for derived verb forms
📎 Question
Is this behavior expected due to internal lexical design choices, or should these cases be considered lemmatization errors in the English model?
Additionally, is there a recommended best practice for handling cases where lemma output is inconsistent across inflected forms of the same verb?
If this behavior is expected and the issue is actually caused by my implementation rather than the English models, I would greatly appreciate it if someone could point that out.
I have made my project publicly available, and I would be grateful if you could take a quick look at the relevant code:
Repository:
Any guidance or suggestions would be greatly appreciated. Thank you for your time and for your work on spaCy.