Skip to content

Fix KDA precision and MD checkpoint correctness - #64

Draft
andresnowak wants to merge 4 commits into
swiss-ai:mainfrom
andresnowak:fix/kda-md-correctness
Draft

Fix KDA precision and MD checkpoint correctness#64
andresnowak wants to merge 4 commits into
swiss-ai:mainfrom
andresnowak:fix/kda-md-correctness

Conversation

@andresnowak

@andresnowak andresnowak commented Aug 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • keep KDA A_log and dt_bias in FP32 through mixed-precision wrapping and support mixed-dtype torch_dist optimizer state
    • Supposedly the idea is that this two values should be in FP32, FlashKDA library also wants them in fp32 (but one can just upcast the values and thats it, that doesn't matter)
    • *But again maybe htis part is okay, we can remove maybe this fix, Like I'm not sure if these two values do really really need fp32 precision on the operations they work on
  • restore KEEL post-layer norms in hybrid decoder specs
  • reject KDA configuration controls that are not implemented (has to do with GDN configs)
  • reserve residual-output handling for actual out_proj weights
  • persist MD preserve-init fixed update norms across ordinary and torch_dist checkpoint resumes
    • Old checkpoints that predate the MD fixed-norm state still load. They recompute the unavailable target from the resumed weight; new checkpoints restore it exactly.

Validation

  • 18 focused regressions passed
  • mixed BF16/native-FP32 optimizer torch_dist round trip passed
  • pre-fix MD torch_dist checkpoint load passed
  • rectangular gains resume matched with maximum parameter difference 0.0
  • ruff check and git diff --check passed

This implementation and pull request were prepared by Codex.

Reject linear-attention controls that KDA does not implement, provide the
KEEL post-layer norms in hybrid blocks, and keep KDA internal projections
out of residual-output projection handling. Add focused regressions for each
case.
Keep tagged KDA decay parameters in fp32 for FlashKDA and map native fp32
parameters alongside fp16/bf16 parameters in sharded optimizer state. Fall
back to the loaded model values when an old checkpoint uses the former master
parameter layout.
Store preserve-init update-normalization targets in optimizer state and project
them through the existing flat-gain sharding path. Version the state with
checkpoint content metadata so pre-fix torch_dist checkpoints still load and
recompute their unavailable targets.
@mac-mvak

Copy link
Copy Markdown
Screenshot 2026-08-31 at 13 42 25 I've also thought of harmonizing bf16 vs fp32 with FlashKDA, I think that's a good idea

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants