Skip to content

Commit 0676ed4

Browse files
Fridah-nvclaude
andcommitted
refactor(export): replay the captured kwargs as they are, cache included
The probe now runs layer_inputs[0] verbatim instead of substituting a fresh past_key_values. A cache only reaches here when the calibration forward loop left caching on, and by then the layer's amax is already wrong -- every replay attended over the keys its own earlier replay wrote. Papering over that in export hides it behind a checkpoint that loads and runs. PR #2248 owns the real fix and warns at the calibration entry point, which is where a user can act on it. Letting the probe raise is the better failure until then: loud, and pointing at the run that produced it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Fridah-nv <201670829+Fridah-nv@users.noreply.github.com>
1 parent b78b331 commit 0676ed4

1 file changed

Lines changed: 0 additions & 8 deletions

File tree

modelopt/torch/export/layerwise_export.py

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -290,14 +290,6 @@ def _fuse_shared_input_scales(self, layer_module: nn.Module, layer_inputs: list
290290
)
291291

292292
args, kwargs = layer_inputs[0]
293-
# A throwaway forward has no use for a cache, and the captured one is not empty:
294-
# calibration already replayed these tuples through it, so attending over it would
295-
# give the probe twice the key length the captured mask was built for. Cache.reset()
296-
# is not an option -- it zeroes the tensors but keeps their length.
297-
if kwargs.get("past_key_values") is not None:
298-
kwargs = dict(kwargs)
299-
kwargs["past_key_values"] = None
300-
301293
input_to_linear, _ = collect_shared_input_modules(
302294
layer_module, lambda: layer_module(*args, **kwargs)
303295
)

0 commit comments

Comments
 (0)