Unable to train model using custom tokenizer

[belalsalih]

How to reproduce the behaviour

Am trying to train a model using a custom tokenizer following the steps in the spacy documentation:
https://spacy.io/usage/linguistic-features#tokenization.
These are the steps that I tried:
Created functions.py as the following:

import spacy
import re
from spacy.tokenizer import Tokenizer
from spacy.util import compile_prefix_regex, compile_infix_regex, compile_suffix_regex

@spacy.registry.tokenizers("custom_tokenizer")
def create_custom_tokenizer():
    def create_tokenizer(nlp):
        infixes = [r"\b\(\b"] + nlp.Defaults.infixes
        infix_re = spacy.util.compile_infix_regex(infixes)
        prefix_re = compile_prefix_regex([r"[-]"] + nlp.Defaults.prefixes)
        suffix_re = compile_suffix_regex([r"[-]"] + nlp.Defaults.suffixes)

        return Tokenizer(nlp.vocab, prefix_search=prefix_re.search,
                                    suffix_search=suffix_re.search,
                                    infix_finditer=infix_re.finditer,
                                    token_match=nlp.tokenizer.token_match,
                                    rules=nlp.Defaults.tokenizer_exceptions)

    return create_tokenizer

Then updated config.cfg as the following:

# tokenizer = {"@tokenizers":"spacy.Tokenizer.v1"}

[nlp.tokenizer]
@tokenizers = "custom_tokenizer"

When I try to run training command I get the following error:

=========================== Initializing pipeline ===========================
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/cli/_util.py", line 92, in setup_cli
    command(prog_name=COMMAND)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/typer/core.py", line 778, in main
    return _main(
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/typer/core.py", line 216, in _main
    rv = self.invoke(ctx)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/cli/train.py", line 53, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/cli/train.py", line 80, in train
    nlp = init_nlp(config, use_gpu=use_gpu)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/training/initialize.py", line 50, in init_nlp
    nlp = load_model_from_config(raw_config, auto_fill=True)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/util.py", line 587, in load_model_from_config
    nlp = lang_cls.from_config(
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/language.py", line 1818, in from_config
    nlp = lang_cls(vocab=vocab, create_tokenizer=create_tokenizer, meta=meta)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/language.py", line 213, in __init__
    self.tokenizer = create_tokenizer(self)
  File "/ssd/ml_all/functions.py", line 21, in create_tokenizer
    token_match=nlp.tokenizer.token_match,
AttributeError: 'English' object has no attribute 'tokenizer'

Not sure if I am missing something here.

Environment
spaCy version 3.6.0
Platform Linux-5.15.0-1041-azure-x86_64-with-glibc2.29
Python version 3.8.10
Pipelines en_core_web_sm (3.6.0), en_core_web_lg (3.6.0)

Thanks.

[shadeMe]

Could you post the complete config and the training command that you're using? Incidentally, you can directly set the prefix_search, suffix_search and infix_finditer properties on the existing Tokenizer instead of reinitializing it (as shown here).

[belalsalih]

training command:
python -m spacy train config.cfg --output ./output_v3 --paths.train ./spacy_models_v3/train_data.spacy --paths.dev ./spacy_models_v3/test_data.spacy --code functions.py

Attached the config.cfg (note: change the file ext. to .txt in order to be able to upload to github)
config.txt
FYR.

Tried your suggestion and got this error:

=========================== Initializing pipeline ===========================
[2023-07-27 10:45:11,262] [INFO] Set up nlp object from config
[2023-07-27 10:45:11,275] [INFO] Pipeline: ['tok2vec', 'ner', 'spancat']
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/cli/_util.py", line 92, in setup_cli
    command(prog_name=COMMAND)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/typer/core.py", line 778, in main
    return _main(
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/typer/core.py", line 216, in _main
    rv = self.invoke(ctx)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/cli/train.py", line 53, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/cli/train.py", line 80, in train
    nlp = init_nlp(config, use_gpu=use_gpu)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/training/initialize.py", line 95, in init_nlp
    nlp.initialize(lambda: train_corpus(nlp), sgd=optimizer)
  File "/ssd/ml_all/.env/lib/python3.8/site-packages/spacy/language.py", line 1318, in initialize
    before_init(self)
  File "/ssd/ml_all/functions.py", line 37, in customize_tokenizer
    nlp.tokenizer.suffix_search = suffix_re #suffix_regex.search
  File "spacy/tokenizer.pyx", line 105, in spacy.tokenizer.Tokenizer.suffix_search.__set__
  File "spacy/tokenizer.pyx", line 621, in spacy.tokenizer.Tokenizer._reload_special_cases
  File "spacy/tokenizer.pyx", line 572, in spacy.tokenizer.Tokenizer._load_special_cases
  File "spacy/tokenizer.pyx", line 615, in spacy.tokenizer.Tokenizer.add_special_case
  File "spacy/tokenizer.pyx", line 565, in spacy.tokenizer.Tokenizer.find_suffix
TypeError: 're.Pattern' object is not callable

Thanks for the quick reply.

[belalsalih]

Apologies, your suggestion did work and I had a mistake in my code.