Downloading leaderboard datasets fails

[sjmonson]

Attempting to download the datasets for leaderboard_v2 fails with ModuleNotFoundError:

$ llm-eval-test download -t leaderboard -d ./datasets
...
Traceback (most recent call last):
  File "/tmp/tmp.VF9s46janQ/venv/bin/llm-eval-test", line 8, in <module>
    sys.exit(eval_cli())
             ~~~~~~~~^^
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/llm_eval_test/__main__.py", line 89, in eval_cli
    datasets = download_datasets(args.datasets, tasks, args.tasks_path, force_download)
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/llm_eval_test/downloader.py", line 20, in download_datasets
    task_dict = tm.load_task_or_group(task_list)
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/lm_eval/tasks/__init__.py", line 415, in load_task_or_group
    collections.ChainMap(*map(self._load_individual_task_or_group, task_list))
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/lm_eval/tasks/__init__.py", line 399, in _load_individual_task_or_group
    group_name: dict(collections.ChainMap(*map(fn, reversed(subtask_list))))
                     ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/lm_eval/tasks/__init__.py", line 314, in _load_individual_task_or_group
    task_config = self._get_config(name_or_config)
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/lm_eval/tasks/__init__.py", line 232, in _get_config
    return utils.load_yaml_config(yaml_path, mode="full")
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/lm_eval/utils.py", line 458, in load_yaml_config
    yaml_config = yaml.full_load(file)
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/yaml/__init__.py", line 105, in full_load
    return load(stream, FullLoader)
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/yaml/__init__.py", line 81, in load
    return loader.get_single_data()
           ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/yaml/constructor.py", line 51, in get_single_data
    return self.construct_document(node)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/yaml/constructor.py", line 60, in construct_document
    for dummy in generator:
                 ^^^^^^^^^
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/yaml/constructor.py", line 413, in construct_yaml_map
    value = self.construct_mapping(node)
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/yaml/constructor.py", line 218, in construct_mapping
    return super().construct_mapping(node, deep=deep)
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/yaml/constructor.py", line 143, in construct_mapping
    value = self.construct_object(value_node, deep=deep)
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/yaml/constructor.py", line 100, in construct_object
    data = constructor(self, node)
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/lm_eval/utils.py", line 442, in import_function
    spec.loader.exec_module(module)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^
  File "<frozen importlib._bootstrap_external>", line 1026, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/lm_eval/tasks/leaderboard/ifeval/utils.py", line 4, in <module>
    from lm_eval.tasks.ifeval import instructions_registry
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/lm_eval/tasks/ifeval/instructions_registry.py", line 17, in <module>
    from lm_eval.tasks.ifeval import instructions
  File "/tmp/tmp.VF9s46janQ/venv/lib64/python3.13/site-packages/lm_eval/tasks/ifeval/instructions.py", line 25, in <module>
    import langdetect
ModuleNotFoundError: No module named 'langdetect'

[sjmonson]

This is due to how lm-eval-harness handles ConfigurableTask objects. When you read a ConfigurableTask, lm-eval imports the associated utility.py which resolves all its imports. Since the leaderboard tasks require additional run time packages we get a ModuleNotFoundError.

A workaround is to install the run time dependencies: pip install llm-eval-test[leaderboard]