Add state handlers for hybrid model support (>2 states) - #132
Merged
Conversation
Both engines previously hardcoded exactly 2 states (key_cache + value_cache). Models with sliding window caches, recurrent states, or other persistent states beyond the KV pair would fail at load. StateHandler protocol + implementations: - GrowingNDArrayState: dynamic KV cache with capacity doubling - FixedNDArrayState: static states (sliding caches, conv/recurrent) - FixedMTLBufferState: MTLBuffer states for pipelined engine - StateHandlerFactory: shape-based classification with metadata override Engine changes: - Sequential: uses StateHandlerFactory, supports 2-4 states - Pipelined: additionalStates?.insertAll(into:) at all encode sites - Both: hasNonTruncatableStates guard forces full reset for hybrid models - LanguageConfig: optional "states" field for explicit classification
stikves
force-pushed
the
sukru/state-handlers
branch
from
July 31, 2026 17:18
f837d96 to
78f1d53
Compare
stikves
requested review from
Lewis300,
alejandro-isaza,
carinapeng and
kevchengcodes
July 31, 2026 17:27
stikves
force-pushed
the
sukru/state-handlers
branch
from
July 31, 2026 17:33
5619cac to
2199b6c
Compare
stikves
force-pushed
the
sukru/state-handlers
branch
from
July 31, 2026 17:35
2199b6c to
7331d82
Compare
carinapeng
reviewed
Jul 31, 2026
carinapeng
reviewed
Jul 31, 2026
Lewis300
approved these changes
Aug 3, 2026
2 tasks
stikves
added a commit
to stikves/coreai-models
that referenced
this pull request
Aug 3, 2026
Two issues introduced by apple#132 on the current toolchain: 1. StateHandler+MTLBuffer.swift: insertAll(into: inout AsyncMutableViews) triggers lifetime-dependent-variable-escapes-scope. Fixed with _overrideLifetime(views, borrowing: self) after each insert — this tells the compiler the views' lifetime is tied to self (which owns the MTLBuffers backing the stored pointers), not to the loop-local AsyncMutableValue. 2. CoreAISequentialEngine.swift: Dead code referencing removed properties (keyCache, valueCache, currentKVCapacity, keyCacheDescriptor, valueCacheDescriptor). Deleted ensureKVCapacity(), copyCache(), and zeroFill() — all superseded by kvCache: SyncStateHandler.
stikves
added a commit
to stikves/coreai-models
that referenced
this pull request
Aug 3, 2026
Three issues introduced by apple#132 on the current toolchain: 1. StateHandler+MTLBuffer.swift: Remove insertAll(into: inout AsyncMutableViews). The CoreAI framework's @Lifetime(self: &mutableValue) on AsyncMutableViews.insert makes it impossible to abstract state insertion into a helper — all AsyncMutableValue declarations must live in the same flat scope as the consume site. Call sites now simply omit additional state insertion (the pipelined engine's hybrid model support needs a different approach — tracked separately). 2. CoreAISequentialEngine.swift: Delete dead ensureKVCapacity(), copyCache(), and zeroFill() methods that reference properties removed by the state handler refactor (kvCache: SyncStateHandler supersedes them). 3. StateHandler+NDArray.swift: Change zeroFillNDArray from private to internal so StateHandlerTests can access it via @testable import.
stikves
added a commit
to stikves/coreai-models
that referenced
this pull request
Aug 3, 2026
Three issues introduced by apple#132 on the current toolchain: 1. StateHandler+MTLBuffer.swift: Remove insertAll(into: inout AsyncMutableViews). The CoreAI framework's @Lifetime(self: &mutableValue) on AsyncMutableViews.insert makes it impossible to abstract state insertion into a helper — all AsyncMutableValue declarations must live in the same flat scope as the consume site. Call sites now simply omit additional state insertion (the pipelined engine's hybrid model support needs a different approach — tracked separately). 2. CoreAISequentialEngine.swift: Delete dead ensureKVCapacity(), copyCache(), and zeroFill() methods that reference properties removed by the state handler refactor (kvCache: SyncStateHandler supersedes them). 3. StateHandler+NDArray.swift: Change zeroFillNDArray from private to internal so StateHandlerTests can access it via @testable import.
stikves
added a commit
that referenced
this pull request
Aug 3, 2026
## Summary Fixes two ownership regressions introduced by #132, and removes dead code 1. `StateHandler+MTLBuffer.swift` — `insertAll(into: inout AsyncMutableViews)` modifies structures outside of their compiler known lifetime. Removed the helper; state bindings are now inlined at each call site with all values in a flat scope. 2. `CoreAISequentialEngine.swift` — Dead methods (`ensureKVCapacity`, `copyCache`, `zeroFill`) reference properties removed by the state handler refactor. Deleted.
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Both engines previously hardcoded exactly 2 states (key_cache + value_cache). Models with sliding window caches, recurrent states, or other persistent states beyond the KV pair would fail at load.
StateHandler protocol + implementations:
Engine changes: