Open
Description
Regression tests in our repo test our recipes with full-size models, and run on a nightly basis. We currently only have a single regression test. This test finetunes Llama2-7B with LoRA for 10 steps then runs our Eleuther eval recipe to check the accuracy of the finetuned model on the truthfulqa_mc2 task #1305.
We should expand coverage of our regression tests across our recipes, and aim to also include tests for memory or performance for some of our models.