BLEU calculation occurs only on the CPU

[LynxPDA]

BLEU calculation occurs only on the CPU, perhaps this is somehow related to the function below

eval.py#L56C44-L56C44

def translator():
    device = "cuda" if ctranslate2.get_cuda_device_count() > 0 else "cpu"
    model = ctranslate2.Translator(ct2_model_dir, device="cpu", compute_type="default")
    if os.path.isfile(sp_model):
        tokenizer = SentencePieceTokenizer(sp_model)
    elif os.path.isfile(bpe_model):
        tokenizer = BPETokenizer(bpe_model, config["from"]["code"], config["to"]["code"])
    return {"model": model, "tokenizer": tokenizer}

I tried replacing device="cpu" with device=device , in which case the GPU was used, but I was getting an OOM error.

[pierotofy]

It makes sense, while using the GPU it probably tries to perform batch translation of all samples at once (which would require a lot of GPU memory).

This would probably require an improvement on OpenNMT to limit the batch size.

[argosopentech]

I've also had some issues trying to get CTranslate2 to run on a GPU.

root@C.7074365:~/Locomotive$ python3 eval.py --config model-config.json 
Starting interactive mode
(en)> Hello this is a test
Traceback (most recent call last):
  File "/root/Locomotive/eval.py", line 239, in <module>
    translation_obj = data["model"].translate_batch(
RuntimeError: Library libcublas.so.11 is not found or cannot be loaded

https://community.libretranslate.com/t/language-model-training-for-argos-translate-lt-locomotive/717/6