v0.0.43

GRAIL v0.0.43 Release Notes

Release Date: December 18, 2025

Highlights

• Delta Checkpointing: Implements sparse delta checkpointing that reduces upload/download bandwidth by ~95% between base checkpoints. Only non-zero weight changes are transmitted instead of full checkpoints, dramatically improving efficiency for trainer-to-miner/validator communication.

• Zstd Compression: Delta files are compressed with zstd for additional bandwidth savings, further reducing transfer times and storage costs.

• Intelligent Upload Strategy: Upload worker automatically selects FULL checkpoints every 10 windows and DELTA checkpoints for intermediate windows, with automatic fallback to FULL uploads on failures to ensure reliability.

• Efficient Weight Reconstruction: Miners and validators download small delta files and reconstruct full model weights locally from cached base checkpoints, significantly reducing download time and bandwidth usage.

• Hash Verification: SHA256 weight hashing ensures bit-exact reconstruction and automatically detects corruption during delta application, maintaining data integrity.

• Enhanced Upload Monitoring: New UploadResult and UploadTiming classes provide granular metrics tracking sparsity ratios, compression ratios, and throughput, with exponential backoff retry logic ensuring reliable checkpoint availability.

v0.0.42

GRAIL v0.0.42 Release Notes

Release Date: December 10, 2025

Highlights

• Fixed MinerValidator Parameter Bug: Removed unexpected download_seconds parameter from MinerResults that was causing validation failures during window processing.

v0.0.41

GRAIL v0.0.41 Release Notes

Release Date: December 9, 2025

Highlights

• Parquet Data Format for Rollout Communication: Introduced Parquet serialization for sharing window inferences/rollouts, reducing data transfer by 40-60% compared to the previous JSON format.

v0.0.40

GRAIL v0.0.40 Release Notes

Release Date: December 8, 2025

Highlights

• Async Training Architecture: Complete refactor to async training with dedicated training_process.py and IPC coordinator for multi-process coordination via shared memory events and queues.

• Checkpoint Management Refactor: Split checkpoint logic into CheckpointPublisher (publishing/retention) and CheckpointManager (consuming), with automatic cleanup of old checkpoints after successful upload.

• Upload Worker: New dedicated process for checkpoint uploads with duplicate prevention per window, resilient subtensor connections, and upload duration tracking.

• Completion Length Gate: Miners now drop groups with short rollouts on their side, improving quality of submitted completions.

• Replay Buffer: Implemented configurable RecencyWeightedBuffer for GRPO training with recency-weighted sampling strategies.

• Memory Efficiency: Added optional gradient checkpointing and optimized logprob computation with autocast and conditional entropy calculation.

• Qwen2.5 Support: Improved chat template format for Qwen2.5 models.

• S3 Pagination: Enhanced list_bucket_files to handle pagination for large result sets.

v0.0.39

GRAIL v0.0.39 Release Notes

Release Date: November 28, 2025

Highlights

• Task Sources with Validation Split: GSM8K and MATH task sources now support train/val/test splits with caching; previously, only train and test were handled.
• MATH Dataset Change: MATH environment now uses the EleutherAI dataset instead of the previous source.
• Cap-Proportional Scoring: Introduced UNIQUE_ROLLOUTS_CAP (5120) for weight allocation! Miners are rewarded based on the fraction of the cap reached rather than just considering the relative performance.
• Burn Percentage: Adjusted GRAIL_BURN_PERCENTAGE from 95% to 80%, giving miners a larger share of the remaining allocation.
• Testing: Added unit tests for GSM8K and MATH task sources covering split handling and caching behavior.

v0.0.38

GRAIL v0.0.38 Release Notes

Release Date: November 25, 2025

Highlights

🔐 Proof Verification Simplified - Consolidated cryptographic proof verification into a single robust sketch-based method, removing rank and histogram checks while maintaining strong security (~10^-117 forgery probability).

Changes

• GRAILVerifier Refactored: Streamlined verification logic to use sketch-only validation (more robust across frameworks)
• Updated Constants: Removed PROOF_MIN_RANK_MATCHES and PROOF_HISTOGRAM_TOLERANCE

🚀 MINERS: REQUIRED UPDATE

⚠️ We are switching to MATH dataset for training. You must update immediately.

git pull && uv sync --all-extras
grail mine

v0.0.37

GRAIL v0.0.37 Release Notes

Release Date: November 14, 2025

Highlights

• MATH dataset environment now live with LaTeX normalization, enhanced answer extraction, and multi-dataset support via unified environment factory and task caching.

• GRPO training efficiency boosted via group filtering and ranking to eliminate low-quality rollouts, plus importance sampling at sequence/token levels and multiple loss variants for algorithm research.

• Offline trainer refined with vLLM-backed logprob verification, improved diagnostics, and TRL baseline script for comparative experiments (moved to research/ directory).

Stability & Quality

• Environment factory pattern simplifies instantiation with cleaner dependency injection and caching.
• Deprecated setup scripts removed (node, local subnet, vLLM environment).

🚀 MINERS: REQUIRED UPDATE

⚠️ We are switching to MATH dataset for training. You must update immediately.

git pull && uv sync --all-extras
grail mine

v0.0.36

📝 Release Note: v0.0.36

Release Date: November 6, 2025

🎯 Primary Highlights

1. Stricter Miner Upload Deadline - Increased enforcement by adjusting DEADLINE_SLACK_SECONDS from 0.5s to -4.0s, ensuring rollouts are uploaded almost a block before the new window starts

2. Smart Rollout Filtering - Added size-based validation to ignore empty/incomplete rollout files (< 200 bytes), improving data quality and reducing processing overhead

✨ Key Features

• Optional KL Loss - KL divergence is now configurable and disabled by default
• vLLM Server Enhancements - Improved vendor-specific parameter support and batch processing for GRAIL proofs
• GRPO Algorithm Refactoring - Consolidated training logic with optional KL controller/tuner
• Improved Monitoring - Enhanced completion length statistics and monitoring logs

v0.0.35

GRAIL v0.0.35 Release Notes

Release Date: November 4, 2025

Highlights

• GRPO algorithm improvements deliver stronger single-window learning via adaptive KL control, better variance filtering, and refined clipping/logprob validation.
• vLLM-backed evaluation now live with unified async server management and configs optimized for high-throughput H200 GPUs.
• Training persistence enhanced with full optimizer/scheduler/RNG state saving and reliable checkpoint recovery after evaluation.

Stability & Quality

• Evaluation configs tuned for GPU memory, max length, and concurrency on H200 hardware.
• Draft offline GRPO trainer added for algorithm experiments (experimental, not production-ready).
• Cleaner logs and WandB metrics aligned to block numbers; CI improvements.

🚀 MINERS: Recommended Update

No protocol changes, but update for minor improved logging and efficiency.

git pull && uv sync
grail mine

---

✅ VALIDATORS: No Action Required

Validation logic unchanged. Optional update for evaluation improvements.

# Optional: Update for observability improvements
git pull && uv sync

v0.0.34

GRAIL v0.0.34 Release Notes

Release Date: October 29, 2025

Highlights

• Validation now challenges ONLY a small subset of completion tokens (not prompts) for logprob & proof checks using indices_from_root_in_range(), noticeably reduced latency without sacrificing security
• Rollout uploads are now enforced 1–2 blocks before window close to block late cheats and keep training windows on schedule.
• Trainer now runs periodic evaluation cycles so every training burst ships with fresh metrics.
• Experimental SGLang and vLLM generation backends are available for testing; they remain unstable and are not yet integrated with the miner loop.
• GSM8K tasks now load once per window (not per problem), eliminating redundant memory overhead.

Stability & Quality

• Checkpoint handling improved with robust file operations (shutil.move)
• Removed verbose debug logging for cleaner production logs
• Python version locked to >=3.10,<3.12 for consistency
• System prompt simplified: removed reasoning length limitation
• Comprehensive test suite for evaluator and loop generation
• Integration tests for block timestamp retrieval

🚀 MINERS: Required Updates

⚠️ ALL MINERS MUST UPDATE — Checkpoint handling streamlined for consistency with validator expectations.

git pull && uv sync
# Restart your miner with the CLI command
grail mine

---

✅ VALIDATORS: No Action Required

Validators do not need updates for this release. Validation logic remains unchanged. However, you may optionally update to benefit from stability improvements:

# Optional: Update for stability improvements
git pull && uv sync