Description
While investigating the perplexity metric, I found that it currently fails when used with GPT-2.
Reproduction
import evaluate
perplexity = evaluate.load("perplexity", module_type="metric")
perplexity.compute(
predictions=["Hello world."],
model_id="gpt2",
)Error 1
On my setup, the metric fails with:
AttributeError: GPT2Tokenizer has no attribute special_tokens_map_extended
The error originates from:
existing_special_tokens = list(tokenizer.special_tokens_map_extended.values())
in metrics/perplexity/perplexity.py.
Error 2
After bypassing that code path, the metric fails with:
ValueError: Asking to pad but the tokenizer does not have a padding token.
The tokenizer is called with:
but GPT-2 does not define a padding token by default.
Environment
- evaluate: 0.4.7.dev0
- transformers: 5.12.1
- datasets: 5.0.0
- torch: 2.12.0+cu130
Notes
GPT-2 is used throughout the perplexity metric examples and documentation, so I would expect the metric to work with GPT-2 out of the box.
I'd be happy to investigate further and submit a PR with a fix.
Description
While investigating the perplexity metric, I found that it currently fails when used with GPT-2.
Reproduction
Error 1
On my setup, the metric fails with:
The error originates from:
in
metrics/perplexity/perplexity.py.Error 2
After bypassing that code path, the metric fails with:
The tokenizer is called with:
but GPT-2 does not define a padding token by default.
Environment
Notes
GPT-2 is used throughout the perplexity metric examples and documentation, so I would expect the metric to work with GPT-2 out of the box.
I'd be happy to investigate further and submit a PR with a fix.