Skip to content

Add ViT MAE pretrain ablation#347

Merged
sgreenbury merged 2 commits intomainfrom
vit-mae-pretrain
Apr 23, 2026
Merged

Add ViT MAE pretrain ablation#347
sgreenbury merged 2 commits intomainfrom
vit-mae-pretrain

Conversation

@sgreenbury
Copy link
Copy Markdown
Contributor

@sgreenbury sgreenbury commented Apr 23, 2026

Summary

  • add CNS deterministic ViT MAE/no-ensemble pretrain preset
  • add timing and 24h MAE Slurm submitters with all progress checkpoints retained and W&B checkpoint artifact logging
  • add MAE-to-CRPS timing and short fine-tune submitters using n_members=16 and batch_size=16
  • document workflow for using MAE checkpoints to seed later CRPS fine-tuning

Validation

  • bash -n slurm_scripts/ablations/vit_mae_pretrain/submit_vit_mae_pretrain_timing.sh
  • bash -n slurm_scripts/ablations/vit_mae_pretrain/submit_vit_mae_pretrain_large.sh
  • bash -n slurm_scripts/ablations/vit_mae_pretrain/submit_vit_mae_to_crps_timing.sh
  • bash -n slurm_scripts/ablations/vit_mae_pretrain/submit_vit_mae_to_crps_large.sh
  • uv run autocast epd --mode local --dry-run datamodule=conditioned_navier_stokes local_experiment=ablations/vit_mae_pretrain/conditioned_navier_stokes/vit_azula_large_mae_no_ensemble logging.wandb.enabled=false trainer.max_epochs=1
  • uv run python -m autocast.scripts.train.encoder_processor_decoder --cfg job datamodule=conditioned_navier_stokes local_experiment=ablations/vit_mae_pretrain/conditioned_navier_stokes/vit_azula_large_mae_no_ensemble logging.wandb.enabled=true logging.wandb.log_model=all +trainer.max_epochs=1
  • uv run autocast epd --mode local --dry-run datamodule=conditioned_navier_stokes local_experiment=epd/conditioned_navier_stokes/crps_vit_azula_large model.n_members=16 datamodule.batch_size=16 +resume_from_checkpoint=/tmp/fake_mae.ckpt +resume_weights_only=true logging.wandb.enabled=true logging.wandb.log_model=all +trainer.max_epochs=1
  • uv run python -m autocast.scripts.train.encoder_processor_decoder --cfg job datamodule=conditioned_navier_stokes local_experiment=epd/conditioned_navier_stokes/crps_vit_azula_large model.n_members=16 datamodule.batch_size=16 +resume_from_checkpoint=/tmp/fake_mae.ckpt +resume_weights_only=true logging.wandb.enabled=true logging.wandb.log_model=all +trainer.max_epochs=1

Add a CNS-only deterministic ViT MAE pretraining preset and matching Slurm timing and 24h submitters.

The run disables ensemble expansion, logs all progress checkpoints, and documents how the checkpoints can seed later CRPS fine-tuning.
Add timing and short fine-tune Slurm submitters for CRPS runs initialized from deterministic MAE checkpoints.

The scripts use n_members=16 with batch_size=16 to preserve the comparison effective global batch while starting a fresh optimizer and time budget.
@sgreenbury sgreenbury merged commit 6a3bdad into main Apr 23, 2026
3 checks passed
@sgreenbury sgreenbury deleted the vit-mae-pretrain branch April 23, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant