`nlp.pipe(n_process>1)` crashes with [E018]/[E871] for any pipeline using floret-mode vectors (e.g. sv_core_news_lg)

[joakimwar]

(🤖 I used Claude Code both to investigate the bug and write this issue)

How to reproduce the behaviour

nlp.pipe(texts, n_process=N) with N > 1 reliably raises [E018] Can't retrieve string for hash '...' (wrapped in [E871] Error encountered in nlp.pipe with multiprocessing) for any pipeline whose vocab.vectors.mode == "floret". It does not happen for pipelines using "default" mode vectors.

This reproduces with a fully public, official spaCy pipeline — no custom training or private data involved:

import spacy

# sv_core_news_lg uses floret-mode vectors (verified: nlp.vocab.vectors.mode == "floret").
nlp = spacy.load("sv_core_news_lg")

texts = [
    "Regeringen presenterade i dag ett nytt forslag om klimatpolitik och energiproduktion.",
    "Fotbollslaget vann matchen med tre mal mot noll efter en stark andra halvlek.",
    "Forskare vid Uppsala universitet har publicerat en ny studie om havsnivahojningar.",
    "Kommunen planerar att bygga fler bostader i centrala delar av staden nasta ar.",
    "Det nya museet oppnar for allmanheten i september med en utstallning om vikingatiden.",
] * 200  # 1000 docs -- enough to fill several multiprocessing batches

if __name__ == "__main__":
    docs = list(nlp.pipe(texts, batch_size=40, n_process=2))
    print(f"OK: processed {len(docs)} docs without error")

Setup:

pip install spacy==3.8.7
python -m spacy download sv_core_news_lg
python repro.py

Result: crashes before completing a single document (well under a second of actual processing time — this is not a rare/late failure, it is immediate and deterministic).

Full traceback

Traceback (most recent call last):
  File "repro.py", line 17, in <module>
    docs = list(nlp.pipe(texts, batch_size=40, n_process=2))
  File ".../spacy/language.py", line 1622, in pipe
    for doc in docs:
  File ".../spacy/language.py", line 1688, in _multiprocessing_pipe
    ...
  File ".../spacy/language.py", line 2410, in _apply_pipes
    byte_docs = [(doc.to_bytes(), doc._context, None) for doc in docs]
  File ".../spacy/util.py", line 1714, in _pipe
    yield from proc.pipe(docs, **kwargs)
  File "spacy/pipeline/transition_parser.pyx", line 245, in pipe
  File ".../spacy/util.py", line 1661, in minibatch
    batch = list(itertools.islice(items, int(batch_size)))
  File ".../spacy/util.py", line 1714, in _pipe
    yield from proc.pipe(docs, **kwargs)
  [... repeated _pipe/minibatch frames through the chained pipeline components ...]
  File "spacy/pipeline/trainable_pipe.pyx", line 75, in spacy.pipeline.trainable_pipe.TrainablePipe.pipe
  File ".../spacy/pipeline/tok2vec.py", line 121, in predict
    tokvecs = self.model.predict(docs)
  File ".../thinc/model.py", line 334, in predict
    return self._func(self, X, is_train=False)[0]
  File ".../thinc/layers/chain.py", line 54, in forward
    Y, inc_layer_grad = layer(X, is_train=is_train)
  File ".../thinc/model.py", line 310, in __call__
    return self._func(self, X, is_train=is_train)
  File ".../thinc/layers/concatenate.py", line 57, in forward
    Ys, callbacks = zip(*[layer(X, is_train=is_train) for layer in model.layers])
  File ".../spacy/ml/staticvectors.py", line 55, in forward
    V = vocab.vectors.get_batch(keys)
  File "spacy/vectors.pyx", line 485, in spacy.vectors.Vectors.get_batch
  File "spacy/strings.pyx", line 185, in spacy.strings.StringStore.as_string
  File "spacy/strings.pyx", line 162, in spacy.strings.StringStore.__getitem__
KeyError: "[E018] Can't retrieve string for hash '90292280375180053'. This usually refers to an
issue with the `Vocab` or `StringStore`."

Root cause (verified experimentally against the source, with an A/B fix test)

(This section has been corrected since the issue was first filed: the original text blamed the Doc.to_bytes()/from_bytes() round trip. Further isolation showed that is not the mechanism — the docs never leave the worker before the crash. The actual cause is Vocab pickling, and a one-line fix has been verified against a source build of current master; see the PR.)

The crash site is the floret string lookup. spacy/ml/staticvectors.py::forward() branches on vocab.vectors.mode:

elif isinstance(vocab.vectors, Vectors) and vocab.vectors.mode == Mode.floret:
    V = vocab.vectors.get_batch(keys)

spacy/vectors.pyx::Vectors.get_batch(), floret branch:

elif self.mode == Mode.floret:
    keys = [self.strings.as_string(key) for key in keys]  # <- raises here

Floret vectors need each token's literal text (not just its hash) to slice character n-grams for the hashing trick, so they look the hash back up in self.strings — the Vectors object's own StringStore reference. When a pipeline is loaded normally, that reference is the same object as vocab.strings (spacy.load / Vocab.__init__ establish this invariant), so every string the tokenizer interns is visible to the floret lookup.

Multiprocessing breaks that invariant. Language._multiprocessing_pipe hands each mp.Process bound methods (self._ensure_doc_with_context and each component's proc.pipe), so under the spawn start method (the default on macOS and Windows) the whole Language/Vocab gets pickled into the worker. And the Vocab pickle path severs the
sharing:

• Vectors.__reduce__ (spacy/vectors.pyx) serializes via to_bytes(), and unpickle_vectors reconstructs the Vectors with its own private, frozen copy of the StringStore — no longer the object vocab.strings points to.
• unpickle_vocab (spacy/vocab.pyx) rewires vocab.strings and vocab.vectors but never re-links them.

In the worker, the tokenizer then interns each incoming token's text into vocab.strings, but the floret lookup consults the severed frozen copy in vectors.strings — so as_string() raises E018 for the first token whose string isn't in the model's stored string table. Immediate and deterministic, exactly as observed. default-mode vectors never call as_string() (they key
rows by hash directly), which is why they're immune.

Each link in this chain was verified in isolation (spaCy 3.8.7 and a source build of current master, 3.8.14):

• After pickle.loads(pickle.dumps(nlp.vocab)), vocab.strings is vocab.vectors.strings flips from True to False.
• Pickling exactly what the worker receives (the bound _ensure_doc_with_context + proc.pipe methods) and running the pipeline in a single process reproduces E018 with the identical failing hash as the multiprocessing run.
• Setting vectors.strings = vocab.strings on the unpickled objects makes the same pipeline run clean.
• Consistent with the pickle mechanism, the bug is spawn-specific:
  mp.set_start_method("fork") (no pickling; Linux's default) runs the full repro to completion on the otherwise-failing setup. Linux users likely never see this, which would explain why it went unreported.

(The originally-suspected mechanism — Doc.to_bytes()/from_bytes() string loss as in #12407 — is ruled out: the traceback fires inside the worker before any result is serialized back, and the worker tokenizes raw text itself, so its vocab.strings demonstrably contains every needed string. Only the vectors' severed copy doesn't.)

Fix (verified)

One line in unpickle_vocab (spacy/vocab.pyx), restoring the invariant that Vocab.__init__ and deserialization already establish:

     cdef Vocab vocab = Vocab()
     vocab.vectors = vectors
     vocab.strings = sstore
+    vocab.vectors.strings = vocab.strings
     vocab.morphology = morphology

A/B-tested against a source build of master (3.8.14): unpatched build fails the repro above with the identical E018/hash; patched build processes all 1000 docs with n_process=2, and the spacy/tests/vocab_vectors + spacy/tests/serialize suites pass (222 passed).

Workarounds until a fix lands (both verified):

1. Force the fork start method before loading the model (POSIX only, with the usual fork-plus-threads caveats): multiprocessing.set_start_method("fork").
2. Re-share the store on unpickle via copyreg — the module must be importable in the workers:

# floret_mp_fix.py — import and call install() before nlp.pipe(n_process>1)
import copyreg
from spacy.vocab import Vocab, pickle_vocab, unpickle_vocab

def unpickle_vocab_fixed(*args):
    vocab = unpickle_vocab(*args)
    vocab.vectors.strings = vocab.strings
    return vocab

def pickle_vocab_fixed(vocab):
    _, args = pickle_vocab(vocab)
    return unpickle_vocab_fixed, args

def install():
    copyreg.pickle(Vocab, pickle_vocab_fixed)

Searched the tracker for prior reports of this specific combination (gh search issues across explosion/spaCy for "floret" "multiprocess", "StaticVectors multiprocessing", "get_batch StringStore") and found nothing — filing this as it appears unreported, despite default-mode vectors clearly working fine under the same n_process codepath.

Impact

Any pipeline trained with floret-mode vectors (a deliberately more compact vector representation, used by several of spaCy's own official non-English _lg pipelines, e.g. sv_core_news_lg) is silently incompatible with nlp.pipe(n_process>1) under the spawn multiprocessing start method (the default on macOS and Windows; Linux defaults to fork and is unaffected unless spawn is selected explicitly) for any component that reaches StaticVectors — which in practice means the parser and/or NER components in the standard _core_news_lg architectures, since they typically share a tok2vec that includes a
StaticVectors layer. The failure is immediate and 100% reproducible, not a rare race — so this should be very easy to confirm, but is also easy to miss until someone actually tries n_process>1 with a floret pipeline (default-mode pipelines never hit it, which is presumably why it's gone unreported).

Your Environment

• Operating System: macOS-15.5-arm64-arm-64bit
• Python Version Used: 3.10.11
• spaCy Version Used: 3.8.7

Info about spaCy

• spaCy version: 3.8.7
• Platform: macOS-15.5-arm64-arm-64bit
• Python version: 3.10.11
• Pipelines: sv_core_news_lg (3.8.0)

Also reproduced the same failure signature against a privately-trained floret-vector pipeline (different language, same spaCy version) before narrowing the repro down to the fully public sv_core_news_lg case above — this is not specific to one model's training data.