Add custom token to the nlp

[quangduc]

Hi,

I want to ask a custom token into parser, so the dependency parser can return different result from the current one. More specifically, here is the current dependency tree:

But Rs is actually the currency of India, so Rs should be quantmod for 10. Can you please tell me how I could guide spaCy to do it without training a new model? Thank you

[adrianeboyd]

Hi, to handle cases like this you want to add a custom norm value for the lexeme (https://spacy.io/api/lexeme#attributes). The norm is used as a model feature to make it easier for the model to generalize across variants like this, also for things like favorite / favourite.

We already do this for some currency symbols by default for all languages, treating them all as $:

spaCy/spacy/lang/norm_exceptions.py

Lines 45 to 58 in 26bf642

	"€": "$",
	"£": "$",
	"¥": "$",
	"฿": "$",
	"US$": "$",
	"C$": "$",
	"A$": "$",
	"₺": "$",
	"₹": "$",
	"৳": "$",
	"₩": "$",
	"Mex$": "$",
	"₣": "$",
	"E£": "$",

So to have a similar entry for Rs, you'd want to modify the lexeme for Rs:

nlp.vocab["Rs"].norm_ = "$"

If you save this model with nlp.to_disk(), these changes are saved as part of the model vocab, so you can reload the custom model without having to make these changes each time.

Some pretrained pipelines have additional language-specific norms, which are available in the package spacy-lookups-data, e.g. for English:

https://github.com/explosion/spacy-lookups-data/blob/544a965501f06f55349e7402e80d6a49bc4cb3cd/spacy_lookups_data/data/en_lexeme_norm.json

[quangduc]

It works line charm. Thanks a lot for your quick reply 👍 .