Complementary to the slow transformer embeddings, we currently use very old static embeddings when using RNN. More recent static embeddings are available and suggest higher performance (quality and speed), see below:

(from https://huggingface.co/blog/static-embeddings)