Skip to content

fix(bigbench): add group for task discovery (bigbench)#3556

Open
jayvenn21 wants to merge 1 commit intoEleutherAI:mainfrom
jayvenn21:fix/bigbench-task-discovery
Open

fix(bigbench): add group for task discovery (bigbench)#3556
jayvenn21 wants to merge 1 commit intoEleutherAI:mainfrom
jayvenn21:fix/bigbench-task-discovery

Conversation

@jayvenn21
Copy link

Fix BIG-bench task discovery

Some BIG-bench tasks exist but are not discoverable under the name bigbench,
causing lm_eval ls tasks and --tasks bigbench to fail with "Tasks not found".

This PR:

  • Adds a group config _bigbench.yaml with group: bigbench listing all 286 BIG-bench subtasks
  • Enables --tasks bigbench and makes "bigbench" appear in lm_eval ls tasks
  • Preserves backward compatibility (no file renames, no task renames)
  • Task list is derived from existing YAMLs under lm_eval/tasks/bigbench/

Fixes #3458

@jayvenn21 jayvenn21 requested a review from baberabb as a code owner February 4, 2026 03:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Tasks not found bigbench

1 participant