GRAIL v0.0.43 Release Notes
Release Date: December 18, 2025
Highlights
-
Delta Checkpointing: Implements sparse delta checkpointing that reduces upload/download bandwidth by ~95% between base checkpoints. Only non-zero weight changes are transmitted instead of full checkpoints, dramatically improving efficiency for trainer-to-miner/validator communication.
-
Zstd Compression: Delta files are compressed with zstd for additional bandwidth savings, further reducing transfer times and storage costs.
-
Intelligent Upload Strategy: Upload worker automatically selects FULL checkpoints every 10 windows and DELTA checkpoints for intermediate windows, with automatic fallback to FULL uploads on failures to ensure reliability.
-
Efficient Weight Reconstruction: Miners and validators download small delta files and reconstruct full model weights locally from cached base checkpoints, significantly reducing download time and bandwidth usage.
-
Hash Verification: SHA256 weight hashing ensures bit-exact reconstruction and automatically detects corruption during delta application, maintaining data integrity.
-
Enhanced Upload Monitoring: New UploadResult and UploadTiming classes provide granular metrics tracking sparsity ratios, compression ratios, and throughput, with exponential backoff retry logic ensuring reliable checkpoint availability.