Read rope_theta from rope_parameters across Inference V2

[alanhuangyoo]

Follow-up to #8341, which covers the same drift in the v1 kernel-injection policy. Separate engine, separate change; #8341 noted this exposure but did not touch it.

What breaks

transformers 5.0 folded the rotary settings into config.rope_parameters and dropped the rope_theta attribute. Eight V2 models still read the attribute.

Seven of them read it directly and raise; exaone4 reads it through a getattr default and quietly uses that default instead:

V2 model	on master (transformers 5.8.0)	with this PR
llama_v2	AttributeError: 'LlamaConfig' object has no attribute 'rope_theta'	10000.0
mistral	AttributeError: 'MistralConfig' …	10000.0
mixtral	AttributeError: 'MixtralConfig' …	1000000.0
phi	AttributeError: 'PhiConfig' …	10000.0
phi3	AttributeError: 'Phi3Config' …	10000.0
qwen_v2	AttributeError: 'Qwen2Config' …	10000.0
qwen_v2_moe	AttributeError: 'Qwen2MoeConfig' …	10000.0
exaone4	silently 1000000.0	10000.0

exaone4 is the worse of the two. Exaone4Config carries rope_parameters['rope_theta'] == 10000.0, so the getattr default puts a 100x rotary base into RotateHalfConfig with nothing raised — wrong frequencies rather than a startup failure.

requirements-inf.txt asks for transformers>=4.32.1 with no upper bound, so 5.x is in range.

The fix

exaone4_5 already reads both spellings — added with the model in #8121:

theta = rope_parameters.get("rope_theta", getattr(config, "rope_theta", None))

Hoist the same lookup onto DSTransformerModelBase as a rope_theta property. DSMoETransformerModelBase extends it, so all eight models are covered by one place and each call site becomes theta_base=self.rope_theta.

The attribute is tried first, so pre-5.0 installs take exactly the path they take today. When neither spelling carries a base it raises instead of guessing one — that is the only behaviour change beyond the fix, and it replaces exaone4's silent 1e6.

exaone4_5 is left alone; its own helper also handles the nested per-layer sliding_attention dict, which is specific to that model.

Test

tests/unit/inference/v2/model_implementations/test_rope_theta.py — the three layouts, the precedence between them, the raise, plus a parametrization over the eight real configs from the installed transformers:

13 passed

tests/unit/inference/v2/model_implementations/   19 passed
yapf --diff / flake8                             clean

I exercised the property and the configs, not a full V2 engine run against downloaded checkpoints.

[alanhuangyoo]

@tohtana — this is the inference/v2 companion to #8341, which you reviewed. Flagging it since it has had no reviewer on it for a week and the two are related.

#8341 is the injection-policy path for Llama; this one is the same rope_theta / rope_parameters move across the eight inference/v2 model implementations that read it. It is independent of #8341 and does not depend on how that one resolves — but if the conclusion there is that scaled rope types have to be refused rather than served, the same question applies here and I would rather hear it once than fix it twice.

[alanhuangyoo]

New information rather than a ping: #8341 merged this morning, and it is the same bug one layer up.

rope_theta moved into config.rope_parameters in transformers 5.x, and getattr(config, "rope_theta", <default>) now silently returns the default for every checkpoint — the value in config.json never reaches the kernel. #8341 fixed that in the AutoTP injection policy (module_inject/containers/llama.py) and @tohtana merged it, so the reasoning has been accepted; this PR is the same fix at a different layer, with no file overlap:

PR	layer
#8341 (merged)	module_inject/containers/llama.py — injection policy
#8345	inference/v2/model_implementations/* — 9 model implementations
#8373	ops/transformer/inference/op_binding/ — the rotary op binding

Merged current master in just now and re-ran on 1×H20; nothing has gone stale.

tests/unit/inference/v2/model_implementations/test_rope_theta.py: 15 passed.

[tohtana]

Hi @alanhuangyoo, thanks for following up, and sorry I missed your earlier ping before merging #8341.

Yes, this fix is still needed, can we apply the same approach regarding RoPE scaling as in #8341?

In #8341, we ended up rejecting unsupported scaled RoPE configurations to avoid silently producing incorrect results. Could you apply the same protection here? The new rope_theta property currently accepts configurations such as rope_type="llama3" and returns only the scalar theta, while these V2 callers do not propagate the additional scaling parameters. Please also add regression coverage for this rejection through both rope_parameters and the legacy rope_scaling layout, while preserving support for unscaled configurations.

[alanhuangyoo]

@tohtana Done in 41a8e41, and no need to apologise — #8341 landing first is what made the shape obvious.

I checked the premise before writing it rather than taking it on faith: every caller of this property does

return RotateHalfConfig(theta_base=self.rope_theta)

and RotateHalfConfig carries use_trained_freqs, theta_base and rotate_dim — nothing else. So a scaled config's parameters have nowhere to go, exactly as you said, and returning the base alone is the silent wrong answer #8341 refused.

What the check reads. _rope_types collects every variant a config asks for across:

• rope_parameters (transformers 5.x) and rope_scaling (4.x)
• both spellings of the key, rope_type and type
• the per-layer-type dicts nested inside either

It runs before the rope_theta attribute read. A 4.x config carrying both rope_theta and a scaled rope_scaling would otherwise return early and never reach the check, which is the case I would most expect in the wild.

Coverage, as requested:

rope_parameters	llama3, linear, dynamic, yarn, longrope
legacy rope_scaling	both key spellings
nested per layer type	a yarn entry beside a default one
attribute present alongside a scaled config	refused rather than short-circuited
unscaled still resolves	absent and "default", in both layouts

Verified they fail for the right reason on the pre-review commit rather than only that they pass now:

9 failed, 17 passed     all nine: Failed: DID NOT RAISE ValueError

Whole directory: master 6 passed, this branch 32 passed (6 + the 26 here), no regressions.

One thing worth your call: this refuses a model that runs today, silently and wrongly. exaone4 in particular used to fall back to 1000000.0 via a getattr default, so a scaled EXAONE-4 config would have been served with unscaled positions rather than refused. I think that is the right trade — it is the same one #8341 made — but it is a behaviour change for anyone who has been unknowingly running one of these, so tell me if you would rather it warn first for a release.