[Feature request] Auto-generate a model card (README.md) for saved speculator checkpoints
Motivation.
I trained DSpark draft models with Speculators and pushed the best checkpoint to the Hugging Face Hub. The upload works, but the repo has no README.md, so the model page is blank: no frontmatter, no link to the verifier, no usage snippet.
Checkpoints currently contain config.json, model.safetensors, optimizer_state_dict.pt, train_command.txt and val_metrics.json, plus run.yaml at the run root — but no README.md, and no model-card or hub-upload code exists in src/ or scripts/.
Why it matters:
- A speculator is useless without its verifier. That link lives only in
speculator_config inside config.json. Putting it in base_model: frontmatter makes the Hub render the relationship.
- Discoverability. Without
library_name/tags, community speculators don't surface in Hub search or filtering.
- The published RedHatAI cards set an expectation the library doesn't help users meet. Hand-writing a card is exactly the step people skip.
- Everything needed is already on disk: architecture from
config.json, training config from run.yaml, command/git SHA/package versions from train_command.txt, validation loss from val_metrics.json. This is templating over existing artifacts.
I couldn't find an existing issue for this — happy to close as a duplicate if I missed one.
Proposed Change.
- Card generator (
src/speculators/model_card.py) that renders README.md from a checkpoint directory. huggingface-hub is already a base dependency, so ModelCard/ModelCardData covers this with no new deps.
- Emit during checkpointing in
BaseCheckpointer.save_checkpoint, so checkpoint_best is directly uploadable. Gated by --model-card / --no-model-card; never overwrite an existing README.md without an explicit flag.
- CLI entry point
speculators model-card <checkpoint_dir> for already-trained checkpoints, reused at the end of speculators convert.
Card contents:
- Frontmatter:
library_name: speculators, base_model: (verifier from config.json), tags: (speculative-decoding, vllm, algorithm), pipeline_tag: text-generation.
- Body: vLLM usage snippet; architecture from
config.json (including algorithm-specific fields like DSpark's markov_rank); training summary from run.yaml; reproduction command and environment from train_command.txt; validation metrics table (below); acceptance.csv/perf_results.csv when present.
Validation metrics table. compute_metrics already produces exactly the numbers a reader wants, and val_metrics.json persists them — they just never reach the card. Render them as markdown:
- A summary row:
accept_len_epoch (expected accepted draft length), accept_rate_epoch, full_acc_epoch, loss_epoch.
- A per-position table from the
position_{i}_acc_epoch keys, one row per draft slot, so readers can see acceptance decay across the block — the single most useful signal for choosing num_speculative_tokens at serve time.
- For DSpark, the confidence-head calibration metrics (
confidence_abs_error_epoch, confidence_cumprod_bias_epoch).
[Feature request] Auto-generate a model card (
README.md) for saved speculator checkpointsMotivation.
I trained DSpark draft models with Speculators and pushed the best checkpoint to the Hugging Face Hub. The upload works, but the repo has no
README.md, so the model page is blank: no frontmatter, no link to the verifier, no usage snippet.Checkpoints currently contain
config.json,model.safetensors,optimizer_state_dict.pt,train_command.txtandval_metrics.json, plusrun.yamlat the run root — but noREADME.md, and no model-card or hub-upload code exists insrc/orscripts/.Why it matters:
speculator_configinsideconfig.json. Putting it inbase_model:frontmatter makes the Hub render the relationship.library_name/tags, community speculators don't surface in Hub search or filtering.config.json, training config fromrun.yaml, command/git SHA/package versions fromtrain_command.txt, validation loss fromval_metrics.json. This is templating over existing artifacts.I couldn't find an existing issue for this — happy to close as a duplicate if I missed one.
Proposed Change.
src/speculators/model_card.py) that rendersREADME.mdfrom a checkpoint directory.huggingface-hubis already a base dependency, soModelCard/ModelCardDatacovers this with no new deps.BaseCheckpointer.save_checkpoint, socheckpoint_bestis directly uploadable. Gated by--model-card / --no-model-card; never overwrite an existingREADME.mdwithout an explicit flag.speculators model-card <checkpoint_dir>for already-trained checkpoints, reused at the end ofspeculators convert.Card contents:
library_name: speculators,base_model:(verifier fromconfig.json),tags:(speculative-decoding,vllm, algorithm),pipeline_tag: text-generation.config.json(including algorithm-specific fields like DSpark'smarkov_rank); training summary fromrun.yaml; reproduction command and environment fromtrain_command.txt; validation metrics table (below);acceptance.csv/perf_results.csvwhen present.Validation metrics table.
compute_metricsalready produces exactly the numbers a reader wants, andval_metrics.jsonpersists them — they just never reach the card. Render them as markdown:accept_len_epoch(expected accepted draft length),accept_rate_epoch,full_acc_epoch,loss_epoch.position_{i}_acc_epochkeys, one row per draft slot, so readers can see acceptance decay across the block — the single most useful signal for choosingnum_speculative_tokensat serve time.confidence_abs_error_epoch,confidence_cumprod_bias_epoch).