Skip to content

Commit b78b331

Browse files
Fridah-nvclaude
andcommitted
fix(export): give the fusion probe no cache, since Cache.reset() keeps its length
The probe replayed the captured kwargs after calling Cache.reset() on the past_key_values they carry. That call does not clear anything: transformers' DynamicLayer.reset() zeroes the key/value tensors "while preserving the objects", so get_seq_length() stays at the captured length and the next update() appends to it. The probe therefore attended over twice the keys the captured attention mask was built for. Invisible in the tests because a tiny unpadded batch reaches the layer with attention_mask=None, where the extra keys only change values the probe discards. A padded batch materializes a 4D mask and the forward raises "The size of tensor a (64) must match the size of tensor b (32) at non-singleton dimension 3". A throwaway forward run only to fire hooks has no use for a cache at all, so pass none. The identical reset() call in the calibration replay predates this branch (#1223) and is PR #2248's to fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com>
1 parent 589be3a commit b78b331

1 file changed

Lines changed: 6 additions & 8 deletions

File tree

modelopt/torch/export/layerwise_export.py

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -290,15 +290,13 @@ def _fuse_shared_input_scales(self, layer_module: nn.Module, layer_inputs: list
290290
)
291291

292292
args, kwargs = layer_inputs[0]
293-
# The tuples were replayed during calibration, so a live cache would give the probe
294-
# twice the key length its mask expects.
295-
cache = kwargs.get("past_key_values")
296-
if cache is not None:
293+
# A throwaway forward has no use for a cache, and the captured one is not empty:
294+
# calibration already replayed these tuples through it, so attending over it would
295+
# give the probe twice the key length the captured mask was built for. Cache.reset()
296+
# is not an option -- it zeroes the tensors but keeps their length.
297+
if kwargs.get("past_key_values") is not None:
297298
kwargs = dict(kwargs)
298-
if hasattr(cache, "reset"):
299-
cache.reset()
300-
else:
301-
kwargs["past_key_values"] = None
299+
kwargs["past_key_values"] = None
302300

303301
input_to_linear, _ = collect_shared_input_modules(
304302
layer_module, lambda: layer_module(*args, **kwargs)

0 commit comments

Comments
 (0)