Out now: spaCy v3.1.0! New pipelines for Catalan & Danish, SpanCategorizer for arbitrary overlapping spans, use predicted annotations during training, bug fixes & more

[ines]

✨ New features and improvements

• NEW: Trained pipelines for Catalan and a new transformer-based pipeline for Danish.
• NEW: Experimental SpanCategorizer component for labeling arbitrary and potentially overlapping spans of text.
• NEW: Use predicted annotations during training via the [training.annotating_components] config setting.
• Alpha tokenization support for Azerbaijani.
• Part-of-speech tag-based lemmatizers for Catalan and Italian.
• The TextCatCNN and TextCatBOW architectures are now resizable.
• Support updating the EntityRecognizer with known incorrect span annotations.
• Auto-generate a pretty README.md based on the meta in spacy package.

For more details, see the New in v3.1 usage guide.

📦 New trained pipelines

Package	Language	UPOS	Parser LAS	NER F
ca_core_news_sm	Catalan	98.2	87.4	79.8
ca_core_news_md	Catalan	98.3	88.2	84.0
ca_core_news_lg	Catalan	98.5	88.4	84.2
ca_core_news_trf	Catalan	98.9	93.0	91.2
da_core_news_trf	Danish	98.0	85.0	82.9

⚠️ Upgrading from v3.0

• Due to the use of configs with extensive versioning, v3.0 pipelines should be compatible with v3.1, however you may see slight differences in performance. Test your v3.0 pipeline with v3.1 against your test suite and if the performance is identical, extend the spacy_version in your model package meta to ">=3.0.0,<3.2.0". If you run into degraded performance, retrain your pipeline with v3.1.
• Use spacy init fill-config to update a v3.0 config for v3.1.
• When sourcing a pipeline component that requires static vectors, it is now required to include the source model's vectors in [initialize.vectors].
• Logger warnings have been converted to Python warnings. Use warnings.filterwarnings or the new helper method spacy.errors.filter_warning(action, error_msg='') to manage warnings.

For more information, see Notes on upgrading from v3.0.

🔴 Bug fixes

• Fix issue ResourceWarning when calling spacy.load in a unittest #7036: Use a context manager when reading model.
• Fix issue Fix scoring normalization #7629: Fix scoring normalization.
• Fix issue ensure 'spacy ray' works #7799: Ensure spacy ray command works.
• Fix issue Show warning if entity_ruler runs without patterns #7807: Show warning if entity ruler runs without patterns.
• Fix issue Fix percent unk display in debug data #7886: Fix unknown tokens percentage in debug data.
• Fix issue Small fix for EL #7930: Make EntityLinker robust for nO=None.
• Fix issue Skip vector ngram backoff if minn is not set #7925: Skip vector ngram backoff if minn is not set.
• Fix issue Fix 'debug model' for transformers + generalize #7973: Fix debug model for transformers.
• Fix issue Preserve existing ENT_KB_ID annotation in NER #7988: Preserve existing ENT_KB_ID in ner annotation.
• Fix issue Handle errors while multiprocessing #8004: Handle errors while multiprocessing.
• Fix issue Fix Docs.from_docs for all empty docs #8009: Fix Doc.from_docs() for all empty docs.
• Fix issue Ensemble textcat with listener #8012: Fix ensemble textcat with listener.
• Fix issue Add ENT_ID and NORM to DocBin strings #8054: Add ENT_ID and NORM to DocBin strings.
• Fix issue Handle partial entities in Span.as_doc #8055: Handle partial entities in Span.as_doc.
• Fix issue Make all Span attrs writable #8062: Make all Span attrs writable.
• Fix issue Update debug data for textcat #8066: Update debug data for textcat.
• Fix issue Custom warning if the doc_bin is too large #8069: Custom warning if DocBin is too large.
• Fix issue Update Vietnamese tokenizer #8099: Update Vietnamese tokenizer.
• Fix issue KB & NEL to/from bytes #8113: Support to/from_bytes for KnowledgeBase and EntityLinker.
• Fix issue Fix offsets in Span.get_lca_matrix #8116: Fix offsets in Span.get_lca_matrix.
• Fix issue Remove unsupported attrs from attrs.IDS #8132: Remove unsupported attrs from attrs.IDS.
• Fix issue ensure tolerance is properly passed on #8158: Ensure tolerance is passed on in spacy.batch_by_words.v1.
• Fix issue Fix bug from Entityruler: ent_ids returns None for phrases #8169: Fix bug from EntityRuler: ent_ids returns None for phrases.
• Fix issue Address missing config overrides post load of models #8208: Address missing config overrides post load of models.
• Fix issue Add all symbols in Unicode Currency Symbols block #8212: Add all symbols in Unicode Currency Symbols to currency characters.
• Fix issue Patterns get duplicated when calling EntityRuler.add_patterns multiple times #8216: Don't add duplicate patterns in EntityRuler.
• Fix issue applying suggestion to avoid mypy errors #8265: Address mypy errors.
• Fix issue Raise error if deps not provided with heads #8335: Raise error if deps not provided with heads in Doc.
• Fix issue Span.lemma joins tokens with spaces unconditionally, which is wrong for languages like Japanese #8368: Preserve whitespace in Span.lemma_.
• Fix issue Don't use the same vocab for source models #8388: Don't clobber vectors when loading components from source models.
• Fix issue Fix non-deterministic deduplication in Greek lemmatizer #8421: Fix non-deterministic deduplication in Greek lemmatizer.
• Fix issue Fix setting empty entities in Example.from_dict #8426: Fix setting empty entities in Example.from_dict.
• Fix issue Add the right return type for Language.pipe and an overload for the as_tuples case #8441: Add correct types for Language.pipe return values.
• Fix issue Various fixes for spans in Docs.from_docs #8487: Fix span offsets and keys in Doc.from_docs.
• Fix issue Fix vectors check for sourced components #8559: Fix vectors check for sourced components.
• Fix issue Raise an error for textcat with <2 labels #8584: Raise an error for textcat with <2 labels.

👥 Contributors

@aajanki, @adrianeboyd, @bodak, @bryant1410, @dhruvrnaik, @explosion-bot, @fhopp, @frascuchon, @graue70, @gtoffoli, @honnibal, @ines, @jacopofar, @jenojp, @jhroy, @jklaise, @juliensalinas, @kevinlu1248, @ldorigo, @mathcass, @meghanabhange, @michael-k, @narayanacharya6, @NirantK, @nsorros, @polm, @sevdimali, @svlandeg, @themrmax, @xadrianzetx, @yohasebe, @ZeeD

---

This discussion was created from the release v3.1.0: New pipelines for Catalan & Danish, SpanCategorizer for arbitrary overlapping spans, use predicted annotations during training, bug fixes & more.