Speed up DALIA export

[haesleinhuepf]

Hi @lea-33 ,

I was just wondering how we could potentially speed up the DALIA export. Do I presume right, that the slow part of the conversion is related to that part?

from transformers import pipeline
model_ckpt = "papluca/xlm-roberta-base-language-detection"
pipe = pipeline("text-classification", model=model_ckpt)
...

Do you think it would be possible to move this part to a separate notebook, that we can run manually from time to time? It could add the language to entries in our yml file where language is not yet defined. In this way, we can cache language, and do not have to run this again and again.

Let me know what you think!

[lea-33]

Hi @haesleinhuepf ,

I checked the workflow again and the problem is not the language detection part, but rather the installation of the dependencies in the workflow (2-3minutes). The execution of the whole script (WITH language detection) takes only a few seconds.

Maybe we could alter the workflow such that it is not triggered with every PR relating to the .yml file, but rather once a month or something?

[haesleinhuepf]

I think if we cache language of all entries in our yml file and update it manually, or once a month like this weekly workflow, this should be fine. That's also more environment friendly than these 3 minutes of compute time (and download traffic) with every pull-request.

[lea-33]

Yes, that makes sense!
I sent a new PR #785 to run the workflow once a month!