[Dataset] [Tool call] Add Hermes function-calling dataset by WindChimeRan · Pull Request #771 · vllm-project/speculators

WindChimeRan · 2026-07-11T01:12:00Z

Purpose

Add the Hermes function-calling dataset (NousResearch/hermes-function-calling-v1, func_calling config) as a predefined preset — a function-calling SFT set (Apache-2.0) originally for post-training.

why:

Tool-use training fixture for the tool-call speculator work, alongside When2Call ([Dataset] [Tool call] Add When2Call single-turn tool-use dataset #769).
Hermes func_calling is the canonical function-calling SFT dataset (used to train Hermes 2 Pro) — genuine tool-call → tool-result → answer trajectories.
License Apache-2.0. Verified no overlap with the nvidia/SPEED-Bench eval set (SPEED-Bench contains no function-calling data; Hermes is not among its 24 sources).

Rebased onto main after #777, the followup that collapsed the two dataset registries into one. This is now a single entry in src/speculators/data_generation/configs.py — since #777 the regeneration script consumes that same registry, so the preset reaches both pipelines (prepare-data and --dataset) by construction, with no change to scripts/response_regeneration/script.py. docs/cli/response_regeneration.md gains the matching row.

Notes for reviewers

The dataset is already in conversations / ShareGPT from,value form (same shape as the existing sharegpt preset), so it needs no normalize_fn — the offline _normalize_conversation maps from/value → canonical roles natively (system → user → assistant → tool → assistant, verified).
Shape, measured over 1299 rows parsed from func-calling.json: each row is a single user request followed by a multi-step agentic trajectory. Raw from sequences: system → human → gpt → tool → gpt (927), system → human → gpt (199), system → human → gpt → tool (163), system → human → tool → gpt (10). 1100/1299 rows carry a tool-result turn; no row has more than one human turn. "Multi-turn" here means the agentic exchange, not multiple user turns.
Tool calls/results are text-encoded (<tool_call> / <tool_response>) inside the turn value, not structured tool_calls fields. Offline training on this is self-consistent (the system turn embeds <tools>; the assistant emits <tool_call> text). The structured tool-aware path ([Regen][tool call] Support tool calls in on-policy response regeneration #750) will additionally need <tool_call> parsing, and must avoid double-injecting the tool schema (the system turn already contains <tools>) — out of scope here.
The tools column is a single JSON string, which _parse_conv_tools consumes via json.loads.
On-policy note. refactor(regen): one dataset registry, every preset usable both on&off-policy #777 makes every registry preset a valid --dataset, so this entry also enables --dataset hermes-fc. That is coherent here: prepare_row() yields [system, user] and the system turn still carries the <tools> schemas, so the model does see the tools and can emit a <tool_call>. The tool-result rounds are dropped, so only the first assistant turn is regenerated; full tool-call regen semantics are [Regen][tool call] Support tool calls in on-policy response regeneration #750. This is the difference from [Dataset] [Tool call] Add When2Call single-turn tool-use dataset #769, which is held as draft: When2Call keeps its tool schemas outside the conversation (a parallel tools column), so on-policy regen there would see no tools at all.

Tests

make quality (ruff check, ruff format, mdformat, mypy): clean.
pytest tests/unit/scripts tests/unit/data_generation: 49 passed.
Schema verified against the real data file — 1299 rows parsed from func-calling.json: id / conversations (from/value) / tools (JSON string) / category / subcategory / task, with <tool_call> and <tool_response> markup present inline. Config/split resolve confirmed against the Hub API (func_calling config → train split, Apache-2.0).
Registry sanity: hermes-fc entry is bare (normalize_fn is None); the offline _normalize_conversation maps a real Hermes tool conversation to system/user/assistant/tool/assistant with the <tool_call> text preserved; prepare_row() on a real row yields [system, user] with the <tools> schemas intact; hermes-fc appears in the --dataset CLI choices with script.py byte-identical to main.

Checklist

I have filled in:

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan/results, such as providing test command and pasting the results.
(Optional) The necessary documentation update.
I (a human) have written or reviewed the code in this pr to the best of my ability.

coderabbitai · 2026-07-11T01:12:09Z

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 8ad90b7e-1b77-47ac-a85e-863143aa8a62

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

📝 Walkthrough

Walkthrough

Changes

Hermes function-calling dataset registration

Layer / File(s)	Summary
Register Hermes function-calling dataset `scripts/response_regeneration/script.py`, `src/speculators/data_generation/configs.py`	Adds `hermes-fc` using `NousResearch/hermes-function-calling-v1`, the `func_calling` subset, and the `train` split; response regeneration also maps prompts from `conversations`.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly matches the main change: adding the Hermes function-calling dataset preset.
Description check	✅ Passed	The description is detailed and directly describes the dataset addition and related validation.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

orestis-z

LGTM. Clean, minimal config addition — follows existing patterns in both registries, and _normalize_conversation already handles the from/value schema (including tool turns) natively. Recommend approving.

🤖 Generated with Claude Code using the /pr-review skill

…f-policy (#777) ## Purpose Dataset-registry half of RFC #702. The regen script kept its own 3-entry dataset dict alongside the shared registry in `data_generation/configs.py`, so adding a dataset means editing both files in two formats (e.g. #769, #771). Unblocked since #716: the script already imports the package. - **Commit 1 (pure refactor):** the script consumes the shared registry; `DatasetConfig` gains `prompt_field`, the bare-prompt fallback column. No behavior change. - **Commit 2:** new `prepare_row()` applies a preset's `filter_fn`/`normalize_fn` the same way prepare-data does, so every registry preset is a valid `--dataset` — datasets work in both pipelines by construction. `--resume` identity and the original three presets' extracted turns are unchanged. Multimodal payloads (sharegpt4v_coco) and tool-call regen semantics are follow-ups. ## Tests - `pytest tests/unit/scripts tests/unit/data_generation`: 48 passed (new: nemotron normalize-parity, filter_fn, map-merge fallback) - `ruff check`, `ruff format --check`, `mypy --check-untyped-defs`: clean - CLI: `--help` offers all presets; `--dataset sharegpt` flows through endpoint auto-detection ## Checklist I have filled in: - [x] The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)". - [x] The test plan/results, such as providing test command and pasting the results. - [x] (Optional) The necessary documentation update. - [x] I (a human) have written or reviewed the code in this pr to the best of my ability. --------- Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com>

mergify · 2026-07-14T14:54:10Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @WindChimeRan.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Register NousResearch/hermes-function-calling-v1 (func_calling config) in the shared dataset registry. Since vllm-project#777 that registry is the single source of presets, so the entry serves both pipelines: off-policy `prepare-data` and the on-policy regeneration script's `--dataset`. Hermes func_calling is a function-calling SFT dataset (Apache-2.0) already in ShareGPT from/value form, so it needs no normalizer (like the existing sharegpt preset). Each row is a single user request followed by a multi-step agentic trajectory: system -> human -> gpt -> tool -> gpt in the common case. Tool calls and results are text-encoded (<tool_call>/<tool_response>) inline; the parallel `tools` column carries the JSON schemas as a JSON string. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LRRGNt2T9ktWyFRboNUZNU Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com>

When2Call (vllm-project#769) is closed, leaving Hermes (vllm-project#771) as the only tool dataset the regenerator targets. Hermes' `tools` column is a JSON string of already OpenAI-wrapped entries (verified: 3582 wrapped / 0 bare / 0 stringified across all 1893 func_calling rows), so the per-entry coercion and the legacy `functions` path had no live consumer. Remove `_normalize_tool` (bare-spec wrapping + per-entry JSON-string decode, only ever needed for When2Call's stringified bare specs) and the `functions` legacy path. `extract_tools` now just reads the `tools` list (decoding a JSON-string column) and keeps the fail-loud guard for a present-but-unusable field. Behaviour is unchanged for Hermes, whose wrapped dicts already passed straight through. Drop the now-dead stringified/bare test; the list-vs-non-list fail-loud cases stay covered. Signed-off-by: Ranran Haoran Zhang <ranzhang@redhat.com>

WindChimeRan force-pushed the add-hermes-fc-dataset branch 2 times, most recently from 731bd12 to a7a2d19 Compare July 11, 2026 01:18

WindChimeRan marked this pull request as ready for review July 11, 2026 01:19

orestis-z reviewed Jul 11, 2026

View reviewed changes

WindChimeRan changed the title ~~[Dataset] Add Hermes function-calling dataset~~ [Dataset] [Tool call] Add Hermes function-calling dataset Jul 11, 2026

WindChimeRan mentioned this pull request Jul 12, 2026

refactor(regen): one dataset registry, every preset usable both on&off-policy #777

Merged

4 tasks

dsikka mentioned this pull request Jul 13, 2026

Q3 Roadmap (WIP) #781

Open

2 tasks

mergify Bot added the needs-rebase label Jul 14, 2026

WindChimeRan force-pushed the add-hermes-fc-dataset branch from a7a2d19 to 49ba4a4 Compare July 14, 2026 15:04

mergify Bot added the documentation Improvements or additions to documentation label Jul 14, 2026

WindChimeRan force-pushed the add-hermes-fc-dataset branch from 49ba4a4 to 65a2b5e Compare July 14, 2026 15:07

mergify Bot removed the needs-rebase label Jul 14, 2026

shanjiaz approved these changes Jul 14, 2026

View reviewed changes

Roderick-Wu approved these changes Jul 14, 2026

View reviewed changes

Merge branch 'main' into add-hermes-fc-dataset

2d72f49

shanjiaz enabled auto-merge (squash) July 14, 2026 18:02

shanjiaz added the ready This PR is ready for review label Jul 14, 2026

shanjiaz merged commit 42a6bee into vllm-project:main Jul 14, 2026
9 checks passed

orestis-z mentioned this pull request Jul 14, 2026

docs(regen): update response regeneration for multi-turn support #766

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Dataset] [Tool call] Add Hermes function-calling dataset#771

[Dataset] [Tool call] Add Hermes function-calling dataset#771
shanjiaz merged 2 commits into
vllm-project:mainfrom
WindChimeRan:add-hermes-fc-dataset

WindChimeRan commented Jul 11, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 11, 2026 •

edited

Loading

Review skipped

Walkthrough

Changes

Uh oh!

orestis-z left a comment

Uh oh!

mergify Bot commented Jul 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

WindChimeRan commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Notes for reviewers

Tests

Checklist

Uh oh!

coderabbitai Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Walkthrough

Changes

Uh oh!

orestis-z left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Jul 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

WindChimeRan commented Jul 11, 2026 •

edited

Loading

coderabbitai Bot commented Jul 11, 2026 •

edited

Loading