Add SyncInputHandler protocol for engine extensibility - #147
Merged
Conversation
stikves
force-pushed
the
sukru/input-handler-protocol
branch
2 times, most recently
from
August 5, 2026 16:16
0168d45 to
438e5d1
Compare
stikves
marked this pull request as ready for review
August 5, 2026 16:16
stikves
force-pushed
the
sukru/input-handler-protocol
branch
2 times, most recently
from
August 5, 2026 18:01
438e5d1 to
f786d80
Compare
tjia1818
reviewed
Aug 5, 2026
stikves
force-pushed
the
sukru/input-handler-protocol
branch
3 times, most recently
from
August 5, 2026 19:09
a2dc47c to
7b614e8
Compare
tjia1818
reviewed
Aug 5, 2026
tjia1818
approved these changes
Aug 5, 2026
stikves
force-pushed
the
sukru/input-handler-protocol
branch
from
August 6, 2026 05:47
e6a33c2 to
f3ce8a3
Compare
Introduces a composable input preparation layer for inference engines: - SyncInputHandler protocol: prepares named NDArray inputs each step - InputContext: carries tokens, position, batch size, sliding window - InputContext.dynamic() for sequential/pipelined engines - InputContext.static() for static-shape (ANE) engines - TokenInputHandler: standard token ID input with batch size caching - CompositeInputHandler: wraps a base handler with extra inputs (RoPE, PLE) - InputCoverage.verify(): fail-fast check at engine init - PipelinedTokenInputHandler: MTLBuffer-based for pipelined engine Also hardens NDArray helpers with stride-aware fill/read: - fillNDArray(count:using:) now checks contiguity and falls back to stride-aware indexing for 4D+ tensors with GPU alignment padding - readNDArray checks contiguity similarly - TokenInputHandler.prepare() validates non-empty batch
stikves
force-pushed
the
sukru/input-handler-protocol
branch
from
August 6, 2026 05:50
f3ce8a3 to
24765f1
Compare
carinapeng
pushed a commit
to carinapeng/coreai-models
that referenced
this pull request
Aug 10, 2026
Static-shape LLM inference engine (StaticShapeEngine). States are discovered by name and the KV cache is right-sized per context bucket (each bucket graph is compiled with its own per-ctx strides, so a max-ctx buffer sliced down corrupts KV); ctx is parsed from the function name. Input preparation extends Sukru's shared SyncInputHandler / InputContext (apple#147): concrete handlers (position ids, causal mask, step, RoPE, PLE, sliding) conform to SyncInputHandler; InputContext gains a per-graph descriptors map so handlers size their own buffers. No Static input protocol/context — one input family for dynamic and static engines. State binding uses bind(into:) (apple#156). Testing: LanguageModelsTests pass incl. new unit tests (ctx-bucket parsing, causal mask fill); correct output on gemma4-E2B, qwen2.5-1.5B, qwen3-0.6B static. Depends on apple#156, apple#147.
msnabiel
added a commit
to msnabiel/coreai-models
that referenced
this pull request
Aug 11, 2026
Resolves conflicts: - Removed ModelShapeConfig.swift (deleted upstream in apple#148) - Updated NDArray+Helpers.swift documentation for stride-aware indexing Pulls in upstream features: - Parakeet speech recognition export and runtime (apple#136) - SyncInputHandler protocol and InputContext (apple#147) - Zero-copy state binding with bind(into:) (apple#156) - Additional speech tests and config improvements - Bug fixes for forced_decoder_ids parsing, topK normalization, logits token count Preserves local iOS optimizations: - int8 KV cache quantization (2× memory reduction) - 16k default context with unreachable shape pruning - CoreAI simulator availability guards 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Composable input preparation layer for inference engines, enabling model-specific inputs (RoPE, PLE, sliding window masks) without engine modifications.
prepare(_ context:) -> [String: NDArray].dynamic()and.static()buildersAlso hardens
NDArray+Helpers:fillNDArray(count:using:)andreadNDArraynow check contiguity and fall back to stride-aware indexing for tensors with GPU alignment paddingTest plan