structures: LTX-2.5 attention forms and host bindings - #174
Open
LiangSu8899 wants to merge 11 commits into
Open
Conversation
LiangSu8899
force-pushed
the
adapt/ltx-25
branch
from
August 16, 2026 10:09
1c55e98 to
531b469
Compare
Map the LTX-2.5 integration onto catalog structures: a sage2 qk-int8/pv-fp8 attention_core backend, a W4A4 NVFP4 vision_ffn backend, the quantization site list as a quantize_on_adopt binding attribute, and a video_generation_pipeline binding. Records the measured qualification context and the sequencing.
attention_core backend on the flashrt/sageattention2-blackwell artifact: INT8 per-warp/per-thread Q/K with FP8 per-channel or FP16 V, bf16 out. Caller-owned workspace allocated once per bound shape, capability envelope read from the artifact, masked and non-128 head-dim sites refuse with the reason on the binder. Measured against the host SDPA at the qualification shapes: 2.3x at the long site with per-call cosine 0.9992 (fp8 variant) / 0.9999 (fp16 variant); real-capture parity sits inside the noise floor established for same-precision kernel substitution.
FP4 speed point of the attention_core family, built on the artifact's fused-prep entry: centering, quantization, delta correction, and attention in one caller-owned workspace. Self-attention only; masked, cross-shape, GQA, non-bf16, and out-of-envelope head dims refuse with the reason on the binder. Accuracy profile is read from the artifact and carried on the module for downstream gates. Pointer-stable call sequences; capture-verified.
Region binding maps the video feed-forward slots on the diffusers host family (zero-bias slots where the host has none; adaptive scale/shift stays outside the seam). Pipeline binding classifies the complete joint audio+video hot path: attention, feed-forward, and whitelisted projections as catalog structures; audio-branch and small-M sites retained on the host with the measured reason; the distilled single-pass recipe, encoder residency, and resident-aware decode tiling declared as binding attributes. Validates under the complete-hot-path contract.
Host family whose processors take separate query/key rotary boundaries, RMS-normalise Q/K after projection, and gate the attention output with per-head sigmoid logits computed from the pre-attention hidden states. The stock Diffusers adapter's processor-state contract does not exist on these modules, so the family gets its own adapter: reproduce the projection half, capture real Q/K/V per called site, bind through the dense attention family, replace only the dispatch. Verified on a joint audio+video block at deployment shapes: attention unit refused by the net-win gate at short sequences and activated 1.257x at long ones, with the audio head-dim refusals legible on the trail. Also treat a hub package's unmet python-dependency declaration (ImportError from the kernels validator) as KernelUnavailable: a host missing a dependency cannot supply the package, and the family ladder falls through instead of aborting the bind.
The two forms bound from a shape-carrying object while the family passes a sequence of per-call capture dicts, and neither was reachable from the family binder -- so they were unreachable code that would have raised the first time a host routed through them. They now qualify a site the way their BF16 siblings do: shape, dtype and mask must hold across the calibration call, an unclaimed shape answers None so the caller keeps its own attention, and an inconsistent calibration or an unserved device raises for the family to record. That walk found a real one: a grouped-query site reached the constructor and raised, where the contract says an unclaimed shape is an answer, not an error. Selection is explicit. bind_dense_attention_best gains a 'prefer' argument, empty by default, so the published order stays precision-first for every host that has receipts against it: these forms trade a bounded numerical error for speed, and that is a decision about a deployment rather than about a device. A caller who has judged the trade names the form and gets the same qualification walk, speed gate and refusal trail as any other rung; an unknown name is an error rather than a silent fallback to the default order. Measured on one transformer block with real weights and real captured inputs, paired inside the gate: at S=24576 the default order reaches 1.15x with the attention family bound but declined at 1.006x, and with sage2 preferred the block reaches 1.49x with the attention unit at 1.257x. Peak memory falls from 12.2GB to 8.2GB when the projections are quantized, and rises to the ceiling of a 32GB part when four attention sites each own a workspace -- pooling those is the open item before the preferred configuration is usable at that size. Cosine against the host is 0.99999 throughout and detach restores it bit-exactly.
LiangSu8899
force-pushed
the
feat/ltx25-structures
branch
from
August 16, 2026 12:03
2615d20 to
0b9e092
Compare
…efault The published attention order is precision-first, so a host that never asked for a quantized form keeps the numerics it has receipts for. That left the forms reachable only by hand, which is the wrong shape for the decision: which executable form may serve a seam, when the trade is a bounded numerical error for speed, is the same kind of statement the scheme already makes about GEMM seams and about the gated-delta and MTP projections. So it arrives the same way. QuantScheme gains attention_forms, empty on every existing profile; the attention adapters that declare scheme awareness read it, exactly as the gated-delta adapter reads its projection format; and 'nvfp4_balance_sage' registers the profile that names them. Naming a form does not force it -- the family still qualifies the shape, speed-gates the result, and falls through to the published order when the installed package does not serve the site. Measured on one transformer block, real weights and captured inputs, paired inside the gate: at S=24576 the projections-only profile reaches 1.15x with the BF16 attention form bound and declined at 1.006x, and the sage profile reaches 1.49x with the attention unit at 1.259x. The difference is the attention kernel's own 45.9 -> 19.6 ms, which is worth naming because a block ratio and a kernel ratio are different numbers for the same 27 ms.
…ging is pooled Two things stood between the quantized attention form and the person who would decide whether to use it. It could only be selected by registering a scheme. The profile is the right home for the decision, but a deployment tuning one seam should not have to author a profile to try it, so attach and auto_swaps take attention_forms directly and it outranks the profile -- the same statement, made by the caller. The answer still comes back measured: the family qualifies the shape, the gate times both arms and prints the accuracy band, and detach reverses it, which is what makes trying it the cheap way to find out. And it did not fit. Each bound seam allocated its own staging and workspace, about 800MB at 24576 tokens over 32 heads, so a transformer whose every block reaches this seam ran out of memory before it could be measured -- which is why the form had only ever been judged one block at a time. The staging is now pooled per (shape, dtype, device, variant, granularity). Sharing is sound because nothing survives a call: every buffer is written at the top of the forward and read before it returns, and the host runs its blocks in sequence on one stream. Held as plain attributes rather than buffers, so a pooled set does not appear in each seam's state_dict. Measured across all 48 blocks at 1536x1024x121f: memory grows by the quantized weights alone, 0.41GB per block, where before each block also took its own workspace. Per-block parity is unchanged with the pool in place -- attention unit cosine 0.999998, whole block 0.999845, band pass throughout.
The measurements a reader needs first are wall clock for one request against the host they would otherwise run, at both sizes. The per-block table stays, relabelled as what it is: a diagnostic that says which family earned which part of that time. The full-size row carries its own qualifications rather than a single ratio: it is eager, the quantized attention profile does not fit at that size yet, and the audio feed-forward stays at host precision because its 126-row calls sit outside the fused chain's alignment. Those three are the distance between this number and a hand-assembled configuration on the same card, and a reader deciding whether to use this should see them next to the number, not after adopting it.
Reverting a routed seam puts the host processor back. It does not free anything: the bound form stays reachable through the plan's observed map and through the very closures that reverted it, and the plan is what a caller holds in order to detach later. So a form the gate declined kept its whole working set for the lifetime of the attachment. Measured on one transformer block at 24576 tokens: the block's own weights fall from 0.720GB to 0.428GB when the projections are quantized, but 0.401GB appeared elsewhere and the attachment came out 0.109GB *heavier* than the host it replaced. Excluding the attention family from the same run leaves 0.009GB unaccounted, which is what identified it. Adapters can now publish a release alongside their revert, dropping their own hold on what they bound; the front door calls it once the gate has settled and no routed unit won. The revert callables survive and stay correct, because an adapter's release empties the route list its closures were built over rather than the closures themselves. The whole-host refusal path already did this through revert_all; what was missing was the mixed outcome, where some units win and the routed ones do not -- which is the ordinary case for this host, since the projections win at every shape and the attention family only wins at long ones. After: the same block attaches at 0.437GB, 0.283GB below the host, with nothing left to reclaim when the plan is dropped. A form the gate activates is untouched, which the sage profile's unchanged 2.045GB confirms.
…t it costs Thirteen catalog specs declare latency.per_shape: true. Nothing has ever read it. The gate takes one measurement, compares one scalar, and records the outcome as though it were universal — while writing the shape it measured into the refusal text, which is the same program stating both that the verdict is local and that it is not. Two recordings, no decision change. min_speedup is untouched, no refusal path is added or tightened, and every verdict this produces is the verdict it produced before. The unit's spec is read, so a refusal from a per-shape rule says it holds for that shape and no other. The shape itself comes from the guards the bound seams were armed with, because m_profile is empty for hosts whose bindings do not declare it and those refusals read "rows unrecorded" — shape-scoped verdicts with no shape on them. And the receipt carries what a unit does to resident memory: what its forms hold, what the host modules they replace hold, and the difference. The walk finds tensors wherever a form keeps them, including inside an artifact's workspace object, counts pooled storage once across sites, and excludes the retained host from its replacement's total. On this host the projections read -0.292 GiB and the declined attention family +0.382 GiB, both agreeing with independent measurement to within 10 MiB. That second number is the point. A refusal has always read as free; this one now reads "no net win (1.005x), +0.38 GiB resident at rows=786432". Whether memory should ever enter the decision is a separate question, and one nobody could argue either way while the data did not exist.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings the LTX-2.5 transformer to the structures layer as an ordinary Diffusers host: two executable forms of
attention_core, the host-family adapter that recognises its attention seam, and the region and pipeline bindings. The model-private runtime is #172; nothing here depends on it.What this adds
adapters/diffusers_gated_rotary_attention.py— the attention seam of hosts whose processors take separate query/key rotary boundaries, RMS-normalise Q/K after projection, and gate the output with per-head sigmoid logits. The stock Diffusers adapter's processor-state contract does not exist on these modules, so it correctly declines them. Recognition is structural: slots, the processor signature, and the declared rope form. No model or class name is discovery evidence.impls/attention_core/sage2_blackwell.py,sage3_blackwell.py— the INT8-QK and FP4 forms, reading their envelope from the installed kernel package. Both qualify through the family's capture convention and answer its way:Nonefor an unclaimed shape, an exception for an inconsistent calibration or an unserved device.bindings/ltx25_dit.yaml,ltx25_video_pipeline.yaml— the region binding for the video feed-forward and complete hot-path coverage for the joint audio+video pipeline: 17 segments, four owned by catalog structures and the rest classified as host stages, state regions or control, with the distilled single-pass recipe, encoder residency and decode tiling budget as binding attributes.Selection is explicit, and the default order does not move
The quantized forms trade a bounded numerical error for speed, which is a decision about a deployment rather than about a device, so they do not enter the published precision-first order that existing hosts hold receipts against. They are, however, meant to be easy to try: the gate measures accuracy and latency and prints both, and
detachreverses the attachment exactly, so trying one is how a deployment finds out rather than something it commits to.They arrive on the axis that already carries such decisions:
QuantSchemegainsattention_forms, empty on every existing profile, read by attention adapters that declarescheme_awareexactly as the gated-delta adapter reads its projection format."nvfp4_balance_sage"registers the profile that names them.attachandauto_swapstakeattention_formsdirectly, which outranks the profile, so tuning one seam does not require authoring a scheme;bind_dense_attention_bestgains the matchingpreferargument for a caller assembling an adapter by hand.The staging is pooled per (shape, dtype, device, variant, granularity). Each bound seam previously allocated its own, about 800MB at 24576 tokens over 32 heads, which is why this form had only ever been judged one block at a time — a transformer whose every block reaches the seam ran out of memory before it could be measured. Sharing is sound because nothing survives a call, and the pool is keyed, pointer-stable and never emptied. Across all 48 blocks memory now grows by the quantized weights alone, 0.41 GB per block, with per-block parity unchanged. Naming a form does not force it — the family still qualifies the shape, speed-gates the result, and falls through when the installed package does not serve the site — and an unknown name raises rather than silently returning the default order's result.
Measured
What a request costs, wall clock, on consumer Blackwell (32GB). The baseline is the unmodified Diffusers host: a 44GB bf16 checkpoint that does not fit on this card, so it runs with weight offloading, which is where a user of this model on this class of hardware actually starts. Same prompt and seed, distilled single-pass recipe, medians of three warm runs, eager.
scheme="nvfp4_balance"scheme="nvfp4_balance_sage"Frames are inspection-equivalent to the host's own output. Attaching is per block, because the bf16 checkpoint is not resident on this card: each block is materialized alone, attached on its own real inputs, and its host weights released before the next.
The full-size row deserves reading closely rather than quoting. All 48 blocks attach, each block's gate measures 1.48×, and the assembled pipeline sits near the card's limit: 23.9 GB resident before the request starts, 28.0 GB at peak, with video-VAE tiling needed for decode to have room. Three things account for the distance between 2.07× and what the same card reaches with a hand-assembled configuration:
Where the time goes
A diagnostic, not the result: which family earned which part of the request time. Real weights, real captured deployment inputs, paired alternating timing inside the gate.
scheme="nvfp4_balance""nvfp4_balance_sage""nvfp4_balance""nvfp4_balance_sage"Matched-forward cosine against the host's own output is 0.99999 in every row;
detachrestores it bit-exactly (max-abs 0.0); ledger fallbacks are zero.Two numbers in that table are easy to misread. A block ratio is not a kernel ratio: the attention this profile selects measures 45.9 → 19.6 ms (2.34×) on its own, and appears as 1.259× for the attention unit because the unit is judged against the whole block — the same 27 ms, a different denominator. And the 1.006× row is the BF16 form's verdict (46.1 ms against the host's 45.9), not a verdict on the quantized ones.
The sage3 form refuses head_dim 128 on a CUDA 12.8 host, because that is what the published package advertises there — its 128-wide build is CUDA 13 only. The ladder falls through and the refusal is on the trail. This repository keeps no second architecture table.
Not changed here
No
structure.yamlbytes, so nospec_digestmoves and no receipt is orphaned. Theattention_corevariant enum does not name these forms; extending it is a version decision left to maintainers, andgates.pycurrently validates variant keys rather than values, which is worth settling in the same bump. Nocsrc, bindings or CMake. The default attention order, and therefore every other host's behaviour, is unchanged.Structures self-review
Scope
attention_coreandvision_ffnare used as published.Contracts and failure modes
Evidence
tests/test_sage_attention_forms.py, 30 cases.Maintenance and hygiene
bind_dense_attention_bestgained a keyword with a default that preserves every existing call.Validation
pytest tests/test_sage_attention_forms.py30 passed; withtests/test_ltx25_contracts.py tests/test_attention_variant_family.py73 passed;tests/test_structures_bindings.py tests/test_structures_attention_logical_dims.py35 passed. Bindings validate underload_binding(require_pipeline_coverage=True).