BLEU: add COCO/PTB tokenization mode (tokenizer_name) for parity with pycocoevalcap#698

Open

bavi404 wants to merge 1 commit intohuggingface:mainfrom

bavi404:fix/bleu-coco-tokenization

Commits on Aug 13, 2025

bleu: add tokenizer_name option with COCO/PTB tokenizer and whitespace tokenizer; add focused tests and docs

Bavishya Sankaranarayanan
authored and
Bavishya Sankaranarayanan
committed