Kind
models
Question
Can we train efficiently on the new coreweave GPU cluster? Do we do better than the best model Eric trained in #75 (eval loss 2.7) if we epoch for 16 epochs instead of 8 and switch to a 3B instead of 1.5B qwen model?
Hypothesis
N/A
Background
We have limited use of a large GPU cluster this week
Approach
Get the data onto coreweave's S3 equivalent. Make sure all objects we create have an initial prefix (path component) of "MarinFold" so that they can be removed later.
Launch a model training sweep using a few variations focusing on ~3B parameter models and similar values for other parameters to what Eric tried in #75
IMPORTANT: Always use iris "batch" priority for all of this.
Success criteria
We trained a model to < 2.7 eval loss
Notes
Kind
models
Question
Can we train efficiently on the new coreweave GPU cluster? Do we do better than the best model Eric trained in #75 (eval loss 2.7) if we epoch for 16 epochs instead of 8 and switch to a 3B instead of 1.5B qwen model?
Hypothesis
N/A
Background
We have limited use of a large GPU cluster this week
Approach
Get the data onto coreweave's S3 equivalent. Make sure all objects we create have an initial prefix (path component) of "MarinFold" so that they can be removed later.
Launch a model training sweep using a few variations focusing on ~3B parameter models and similar values for other parameters to what Eric tried in #75
IMPORTANT: Always use iris "batch" priority for all of this.
Success criteria
We trained a model to < 2.7 eval loss
Notes