[DRAFT][GAUDISW-248216] Flag compact-GDN per-layer states as independent for the Synapse bridge - #1563
Draft
moshehoori wants to merge 1 commit into
Draft
Conversation
… the Synapse bridge Compact GDN mode packs many logically-independent per-layer recurrent-state tensors into one shared `torch.zeros` allocation, so they all report the same `storage().data_ptr()`. The Synapse bridge keys its duplicate/alias bookkeeping on that storage base and mis-binds the siblings (first-wins), corrupting decode output (intermittent gsm8k=0 / "!!!!" token spam on Qwen3.5-397B-A17B-FP8, TP=4 Gaudi3). Tag each compact GDN state tensor's backend meta with `is_independent_packed_state` so the bridge keys those tensors by per-tensor data_ptr (base+offset) instead of the storage base, giving each its own section while keeping a tensor's own in-place input<->output bound. The bridge honours the tag only when `PT_HPU_GDN_COMPACT_INDEPENDENT_TENSORS=1`, and the helper silently no-ops if the running bridge lacks the binding, so this is inert on stacks that predate the bridge-side change. Companion to the pytorch-integration change that adds the flag + binding. Hardware A/B (Qwen3.5-397B + TP=4): flag-on 6/6 runs clean; flag-off fires. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
moshehoori
had a problem deploying
to
pre-merge-approval
June 25, 2026 13:30 — with 
GitHub Actions
Error
GitHub Actions
Error
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a small helper in the HPU v1 model runner to tag compact-GDN state tensors as “independent packed state” via Habana’s tensor extra-meta, so the Synapse bridge can key section bindings by per-tensor data_ptr (base+offset) rather than storage base when enabled on the bridge side.
Changes:
- Introduces
_mark_independent_packed_state()to setmeta.is_independent_packed_state = True(best-effort). - Calls the helper for each compact GDN state tensor allocated in
initialize_kv_cache.
|
|
||
|
|
||
| def _mark_independent_packed_state(tensor: torch.Tensor) -> None: | ||
| """GAUDISW-248216: tag a compact-GDN state tensor as logically independent. |
Comment on lines
+628
to
+640
| try: | ||
| from habana_frameworks.torch import _core_C | ||
| except Exception: | ||
| return | ||
| try: | ||
| try: | ||
| meta = _core_C.get_new_tensor_extra_meta(tensor) | ||
| except RuntimeError: | ||
| meta = _core_C.get_tensor_extra_meta(tensor) | ||
| if meta is not None: | ||
| meta.is_independent_packed_state = True | ||
| except Exception as e: # pragma: no cover - defensive | ||
| logger.warning("Failed to mark independent packed GDN state: %s", type(e).__name__) |
Comment on lines
+6104
to
+6110
| # GAUDISW-248216: compact mode packs every GDN layer's | ||
| # per-request state into ONE shared allocation, so many | ||
| # logically-independent state tensors report the same | ||
| # storage().data_ptr(). The Synapse bridge keys its | ||
| # duplicate/alias bookkeeping on that storage base and | ||
| # would mis-bind the siblings. Flag each compact state | ||
| # tensor as independent so the bridge keeps them apart. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GAUDISW-248216 — flag compact-GDN states as independent for the Synapse bridge
Problem
Compact GDN mode packs many logically-independent per-layer recurrent-state tensors into one shared
torch.zerosallocation, so they all report the samestorage().data_ptr(). The Synapse bridge keys its duplicate/alias bookkeeping on that storage base and mis-binds the siblings (first-winsemplace) — one layer's state gets bound to another's section at launch, corrupting decode output: intermittentgsm8k=0/!!!!token spam on Qwen3.5-397B-A17B-FP8, TP=4 Gaudi3 (~17–67% of fresh skip-warmup compiles).Change
Tag each compact GDN state tensor's backend meta with
is_independent_packed_state, via a small best-effort helper called at the compact-allocation site ininitialize_kv_cache. This tells the Synapse bridge to key those tensors by per-tensordata_ptr(base+offset) instead of the storage base — giving each sibling its own section while keeping a tensor's own in-place input↔output bound.PT_HPU_GDN_COMPACT_INDEPENDENT_TENSORS=1(default off).Companion change
Requires the pytorch-integration change that adds the
is_independent_packed_statemeta bit, the_core_Cbinding, the env flag, and theget_section_key()bridge logic.Validation
Hardware A/B (Qwen3.5-397B-A17B-FP8, TP=4, lm_eval gsm8k):
!!!!firing🤖 Generated with Claude Code