Update Evaluation Logic to Latest `lm_eval` (0.4.8) and Support Automatic Benchmark Evals w/o Validation Set#1348

Open

Kyle1668 wants to merge 12 commits intomainfrom

Commits on May 9, 2025

Update eval code

Kyle1668
authored and
Quentin-Anthony
committed
Resolve W&B error

Kyle1668
authored and
Quentin-Anthony
committed
Add eval results to gitignore and specify latest lm_eval in reqs file

Kyle1668
authored and
Quentin-Anthony
committed
Add limit neox arg

Kyle1668
authored and
Quentin-Anthony
committed
Update ci to Python 3.9

Kyle1668
authored and
Quentin-Anthony
committed
Resolved challenge with task groups

Kyle1668
authored and
Quentin-Anthony
committed
Update docs

Kyle1668
authored and
Quentin-Anthony
committed
Make type handling clearer for task groups

Kyle1668
authored and
Quentin-Anthony
committed
Run eval tasks even when no validaiton set is provided

Kyle1668
authored and
Quentin-Anthony
committed
Removed commented out code

Kyle1668
authored and
Quentin-Anthony
committed
Run pre-commit

Kyle1668
authored and
Quentin-Anthony
committed
Resolved potential transformer engine bug

Kyle1668
authored and
Quentin-Anthony
committed