Skip to content

Conversation

NathanHB
Copy link
Member

@NathanHB NathanHB commented Oct 13, 2025

getting rid of the default_tasks.py file containing 1000s tasks and putting all of them into separate files. This allows for:

  • better maintenance
  • more readability (one task one file)
  • adding metadata for every tasks

Also, getting rid of many duplicate tasks, for example, the same task in 2 different suites. I'm only keeping/moving all tasks to the lighteval suite so that we can eventually get rid of the suite system.

  • move tasks from default_tasks
  • move extended tasks
  • move multilingual tasks
  • move community tasks
  • add docstring for default tasks
  • add docstring for multilingual tasks
  • add docstring for community tasks
  • make a space to explore all the tasks, based on the tasks docstrings.

We can then use the metadata and the one file system to build a viewer for all the tasks:

Screenshot 2025-10-14 at 17 03 23

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant