hellaswag_arabic_pfn uses eval() instead of ast.literal_eval() on a dataset field, arbitrary code execution from an untrusted HF dataset #1293

Bug Description

hellaswag_arabic_pfn in src/lighteval/tasks/multilingual/tasks/arabic.py (the prompt function for the hellaswag_okapi_ar task) parses a dataset field with the builtin eval() instead of ast.literal_eval():

def hellaswag_arabic_pfn(line, task_name: str = None):
    ctx = re.sub(r"\[.*?\]", "", line["ctx"])
    endings = [
        re.sub(r"\[.*?\]", "", e) for e in eval(line["endings"])
    ]  # endings is a string representation of a list

line["endings"] comes directly from the hf_repo="OALL/AlGhafa-Arabic-LLM-Benchmark-Translated" dataset on the Hub, a community-hosted dataset outside lighteval's own control. Since eval() executes arbitrary Python, not just list literals, a crafted endings field in that dataset (or any future revision of it, or a similarly-named dataset someone points a custom task at) can run arbitrary code the moment this task is loaded and evaluated, i.e. on whatever machine is running the benchmark.

Steps to Reproduce

import ast

legit = "['ending one', 'ending two', 'ending three']"
print(eval(legit))              # ['ending one', 'ending two', 'ending three']
print(ast.literal_eval(legit))  # same result, safely

malicious = "__import__('os').system('id > /tmp/pwned_by_dataset.txt')"
eval(malicious)  # executes the shell command, file gets created

Confirmed locally: eval() on the malicious payload actually ran the injected shell command and wrote the file, while ast.literal_eval() on the exact same payload safely raised ValueError: malformed node or string instead of executing anything, and still returns the identical result for the legitimate case.

Expected Behavior

This should use ast.literal_eval(), which is in fact the pattern lighteval already uses everywhere else for this exact situation, string-encoded list fields coming from HF datasets:

• src/lighteval/tasks/tasks/sacrebleu.py:39 — line["translation"] = ast.literal_eval(line["translation"])
• src/lighteval/tasks/tasks/race_high.py:33 — line["problems"] = ast.literal_eval(line["problems"])
• src/lighteval/tasks/tasks/musr.py:39 — choices = ast.literal_eval(line["choices"])
• src/lighteval/tasks/multilingual/tasks/swiss_legal/main.py:366 — same pattern

arabic.py:603 is the one place using raw eval() for what looks like the identical use case, so this looks like a one-off inconsistency rather than an intentional choice, and the fix is a straightforward drop-in replacement (ast.literal_eval handles list/string/number literals identically to eval for well-formed input, it just refuses anything that isn't a literal).

Affected code

src/lighteval/tasks/multilingual/tasks/arabic.py, line 603, inside hellaswag_arabic_pfn.

System

huggingface/lighteval main branch.