[CI] [JIT] [Build] [Bugfix] Build pybind modules against torch's bundled pybind11; drop pip pybind11 dep

[Rohan138]

Problem

aiter's JIT builder (aiter/jit/utils/cpp_extension.py) appends the pip-installed pybind11's include path via pybind11.get_include(), emitted as a -I ahead of torch's bundled headers (-isystem <torch>/include). So every compiled pybind module resolves #include <pybind11/...> to whatever pybind11 pip has.

pybind11 3.1.0 (2026-08-06) bumped PYBIND11_INTERNALS_VERSION 11 → 12. Modules built at different pip epochs then disagree on the internals version, and pybind11 does not share registered types across internals versions. The prebuilt module_aiter_core (which registers aiter_tensor_t) and a JIT module built later end up incompatible:

TypeError: fmha_fwd_bf16_opus_fwd(): incompatible function arguments.
  Invoked with: <aiter.jit.module_aiter_core.aiter_tensor_t object at ...>

Fixes #4770.

Root cause detail

module_aiter_core is a torch-free (torch_exclude:True) pybind module — it registers aiter_tensor_t and is loaded via ctypes. It has no torch headers on its include path, so it previously took its pybind headers only from pip. Torch-backed consumer modules (e.g. fmha) also took pip's. Because they were compiled at different times, a drifting pip pybind11 desynchronized producer and consumers.

Fix

torch already bundles pybind11 at torch/include (3.0.1 / internals v11 in torch 2.10), which satisfied aiter's former pybind11>=3.0.1. This PR builds every python module — torch-backed and torch-free — against torch's bundled pybind headers:

• Drop the pip get_include() append entirely.
• For torch_exclude builds (which otherwise omit torch/include), add torch/include back only for header resolution — pybind is header-only, so the torch-free .so gains no runtime libtorch dependency.

Now producer (module_aiter_core) and all consumers compile against the same pybind (torch's), so aiter_tensor_t stays ABI-compatible regardless of any pip pybind11. (Vendoring pybind instead would re-break this: torch-backed modules must use torch's pybind to interop with torch types, so everything has to converge on torch's copy.)

With the ABI sourced from torch, aiter no longer needs a pip pybind11 dependency at all. Removed from:

• requirements.txt, pyproject.toml (build-system.requires), setup.py (install_requires)
• the pip install --upgrade pybind11 steps in CI workflows (aiter-test, atom-test, kimi-downstream, kimi-perf-downstream, vllm_benchmark) and .github/requirements/triton-test.txt

Test

rocm/pytorch:latest (torch 2.10.0+rocm, bundled pybind v11), with no pip pybind11 installed, building trivial pybind modules through aiter's load():

Module	is_python_module	torch_exclude	.so internals	links libtorch?
torch-backed	True	False	__pybind11_internals_v11__ ✅	yes (expected)
torch-free (like module_aiter_core)	True	True	__pybind11_internals_v11__ ✅	no ✅

Both compile with zero pip pybind11 present. For contrast, with pip install pybind11==3.1.0, the pre-change code produced __pybind11_internals_v12__ (the mismatch); this change yields v11 regardless.

[github-actions]

🏷️ CI Guide

Runs automatically on every PR:

• ✅ Pre-checks (submodule verification, code formatting)
• ✅ Aiter op tests (gfx942 + gfx950)
• ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label	Tests
ci:gfx1250-ffm-triton	Run the five-shard gfx1250 FFM Triton test suite
ci:triton-300x	Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang	SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom	ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full	ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm	vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all	All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4996 --add-label <label>

PR title tags:
Component tags ([Triton/Gluon], [HIP], [CK], [ASM], ...) are added to the PR title automatically from the changed files and re-synced on every push — change-type tags like [fix]/[Perf] and op tags like [MLA] are left untouched. Add the no-auto-title label to opt this PR out of title tagging.

[Copilot]

Pull request overview

This PR fixes pybind11 ABI incompatibilities between prebuilt and runtime-JIT’d aiter pybind modules by ensuring all pybind11 headers resolve to torch’s bundled copy (torch/include), and removes the repository’s pip pybind11 dependency accordingly.

Changes:

• Stop adding pip-installed pybind11 include paths during JIT extension builds; ensure torch/include is available for header resolution even under torch_exclude python-module builds.
• Remove pybind11 from packaging metadata (requirements.txt, pyproject.toml, setup.py) now that torch’s bundled pybind11 is the single source of headers.
• Remove CI steps that explicitly install/upgrade pybind11 in downstream and test workflows.

Reviewed changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
aiter/jit/utils/cpp_extension.py	Ensures pybind11 headers come from torch/include for all python modules (including torch_exclude) to prevent internals-version mismatches.
setup.py	Drops pybind11 from install_requires.
requirements.txt	Drops pinned pybind11 requirement.
pyproject.toml	Drops pybind11 from build-system.requires.
.github/workflows/aiter-test.yaml	Removes CI installation/upgrade of pybind11.
.github/workflows/atom-test.yaml	Removes CI installation/upgrade of pybind11.
.github/workflows/kimi-downstream.yaml	Removes CI installation/upgrade of pybind11.
.github/workflows/kimi-perf-downstream.yaml	Removes CI installation/upgrade of pybind11.
.github/workflows/vllm_benchmark.yaml	Removes CI installation/upgrade of pybind11.
.github/requirements/triton-test.txt	Removes pybind11 from triton-test requirements.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Rohan138]

The fix in this PR (using the pytorch-bundled pybind11) does tie AITER to the corresponding PyTorch version it was built with, but I assume that's already the case for now. Might need to be revisited if we want to make AITER PyTorch-stable.

Also I don't know if the deps should be pinned here in the first place: https://github.com/ROCm/aiter/blob/main/requirements.txt

Or at the very least, the build and install deps should be kept aligned: https://github.com/ROCm/aiter/blob/main/setup.py#L468

If the deps need to be pinned for CI/test purposes, it might be better to have a separate tests/requirements.txt file the way we do in vLLM: https://github.com/vllm-project/vllm/blob/main/requirements/test/rocm.txt; the pins right now, if copied over to setup.py will e.g. break vLLM CI builds if vLLM bumps/pins a different pandas versions