Skip to content

pip install ships task configs without their extensionless _*_yaml include files → FileNotFoundError breaks all task loading #1350

@wl3181-bit

Description

@wl3181-bit

Checklist

  • I have searched for similar issues before opening this one.
  • I am using the latest version of lmms-eval.

Bug Description

After pip install lmms-eval, every evaluation run fails during task registration. TaskManager.initialize_tasks() scans all task directories at startup; when it reaches screenspot it crashes:

FileNotFoundError: [Errno 2] No such file or directory:
'.../site-packages/lmms_eval/tasks/screenspot/_default_template_rec_yaml'

Because the scan is global, this kills runs for completely unrelated tasks too (e.g. mmmu_val) — you can't evaluate anything from a pip-installed lmms-eval.

Steps to Reproduce

pip install lmms-eval==0.7.1
lmms-eval --tasks list

Error Message / Traceback

FileNotFoundError: [Errno 2] No such file or directory: 
'.../site-packages/lmms_eval/tasks/screenspot/_default_template_rec_yaml'

Environment

  • lmms-eval 0.7.1, installed from the PyPI wheel
    • Python 3.10

Additional Context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions