Skip to content

Releases: vllm-project/speculators

Speculators v0.2.0

03 Nov 15:10
@dhuangnm dhuangnm
v0.2.0
02212fa
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Speculators v0.2.0 Latest
Latest

Speculators v0.2.0 Release Notes

This Speculators v0.2.0 release introduces the following new features and enhancements:

  • Support for Draft Models with Multiple Decoder Layers: Previously, only draft models with a single decoder layer were supported. The Eagle3 converter now sets the num_hidden_layers from the config instead of always assuming one layer.
  • Added Support for eagle_aux_hidden_state_layer_ids Argument: This new argument allows users to toggle the layer IDs of the hidden state layers that are fetched during inference time. This enables support for converting Llama4 Maverick draft models to the Speculators format and running the converted model in vLLM.

Updates and Deprecations:

  • Python 3.9 Support Removed: Support for Python 3.9 has been removed and will no longer be provided. Python 3.10+ will be supported going forward.
  • Default Number of Speculative Tokens Changed: The default number of speculative tokens has been changed from 5 to 3 for all Eagle and Eagle3 models.
  • Override tie_weights() in Eagle3Speculator: This override prevents vocabulary corruption and supports Transformers 4.54.1.
  • Updated head_dim Calculation in Eagle3 Converter: The head_dim value is now used from the config if provided; otherwise, it is calculated using the formula hidden_size // num_heads.
  • Eagle3 Draft Models Retain Original Dtype: All Eagle3 draft models now keep their original dtype after being converted to the Speculators format. Previously, all converted draft models were cast to FP32.
  • Extended Logic for target_vocab_size: The system defaults to using the "t2d" length, but if not available recursively search the verifier model's config file for vocab_size.
  • Full End-to-End vLLM Smoke Testing: Extended and added full end-to-end vLLM smoke testing for both converted and unconverted models.

Full Change Log

New Contributors

Full Changelog: v0.1.0...v0.2.0

Contributors

dbarbuzzi, eldarkurtic, and 8 other contributors
Assets 4
Loading

Speculators v0.1.0 -- First Public Release

08 Aug 01:45
@markurtz markurtz
v0.1.0
8a49095
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

Choose a tag to compare

View all tags
Speculators v0.1.0 -- First Public Release

Overview

This first public release publishes the complete initial codebase for Speculators — a unified library for building, evaluating, converting, and serving speculative decoding algorithms for LLMs. It delivers the core framework, CI/CD and developer workflow, model/config implementations (EAGLE v1/HASS/EAGLE‑3), converter CLIs from external research repos, a Hugging Face–compatible model format with vLLM serving support, and prototype training code.

What’s New (Highlights)

  • Unified, extensible framework for speculator models (build, evaluate, convert, store)
  • Hugging Face–compatible speculator format with serving support landed in vLLM
  • Models/configs for EAGLE v1 (HASS-style), HASS, and EAGLE‑3 (multi-layer types)
  • Checkpoint converter CLIs (Eagle, Eagle‑3) from external research repositories
  • Prototype training code and scripts (EAGLE‑1-style drafter, HASS) + requirements
  • Production readiness: CI/CD, tests, style, docs, examples, and benchmarks

Use Cases Enabled

  • Register and configure new speculator algorithms via a standardized configuration and registry system
  • Convert external checkpoints (EAGLE/EAGLE‑3/HASS variants) into the Speculators format with CLI tools
  • Serve Speculators models directly in vLLM for low‑latency inference
  • Evaluate and benchmark speculators (e.g., with GuideLLM), including quantized verifier swaps
  • Prototype‑train drafters using provided research code and scripts

Getting Started

  • Install (Python 3.9–3.13 on Linux or macOS): 
    pip install git+https://github.com/neuralmagic/speculators.git
  • Serve with vLLM (requires v1 API): 
    VLLM_USE_V1=1 vllm serve RedHatAI/Qwen3-8B-speculator.eagle3
  • Explore examples and research: examples/, research/eagle3/, research/hass/

Compatibility Notes

  • Python: 3.9–3.13
  • OS: Linux and macOS
  • Transformers pinned to avoid mypy regressions (PR #73)
  • vLLM v1 API required for serving (set VLLM_USE_V1=1)

Full Changelog (v0.1.0)

First public release of Speculators. This release publishes the complete initial codebase and enables the first set of core use cases for speculative decoding with LLMs.

Added

  • Base configuration and registry system with tests: Speculator, Token Proposal, and Model Speculator configs; EagleSpeculatorConfig for EAGLE v1/HASS; config serialization/loading (PRs #26, #27, #28, #29, #34, #36)
  • Eagle speculator model and support for multiple transformer layer types (PRs #37, #49)
  • Eagle‑3 speculator model and Qwen support (PRs #50, #55)
  • Checkpoint converter CLIs: Eagle and Eagle‑3; standardized converter interface (PRs #39, #53, #72)
  • vLLM serving documentation and Qwen benchmark assets (PRs #77, #78, #82, #83)
  • Examples directory and README for getting started (PR #81)
  • Branding assets (icons, logos, user‑flow diagrams) (PR #87)

Changed

  • Standardized converter CLI UX and flags (PR #72)
  • Documentation/readme formatting and content updates (PRs #70, #75, #83, #85)

Fixed

  • Missing embeddings in converted checkpoints/workflows (PR #65)
  • CLI flags and norm_before_residual toggle (PRs #57, #58)
  • Compatibility: pin transformers to resolve mypy/typing regressions (PR #73)

CI/CD and Tooling

  • GitHub Actions: migrated link checks to lychee and updated workflows (PRs #3, #45)
  • PR comment behavior refinements (PR #47)

Research and Training

  • Training code for EAGLE‑1‑style drafter with multi‑step training (PR #35)
  • HASS/EAGLE‑3 research updates, requirements, and DeepSpeed dependency (PRs #64, #67, #69)

Documentation

  • vLLM serving instructions, Qwen benchmark results, examples README, and research readmes (PRs #64, #70, #77, #78, #81, #83, #85)

New Contributors

Thanks also to continuing contributors: @markurtz, @rahul-tuli, @dsikka

Links

Contributors

markurtz, dsikka, and 4 other contributors
Loading