Commit 6265330
Carina Peng
Add static-shape inference engine
Static-shape LLM inference engine (StaticShapeEngine) built on the bind(into:)
state binding from #156. States are discovered by name and the KV cache is
right-sized per context bucket (each bucket graph is compiled with its own
per-ctx strides, so a max-ctx buffer sliced down corrupts KV); ctx is parsed
from the function name. Input preparation is pluggable per model family via
StaticInputProvider + StaticModelProfile; StaticInputContext composes the shared
InputContext and the provider mirrors SyncInputHandler.
Testing: LanguageModelsTests (317) pass; correct output on gemma4-E2B,
qwen2.5-1.5B, qwen3-0.6B static assets.
Known limitation: cross-bucket cache re-layout needs a channel-interleave-aware
copy; within-bucket generation is correct.
Depends on #156.1 parent 1677713 commit 6265330
4 files changed
Lines changed: 827 additions & 180 deletions
File tree
- swift/Sources/CoreAILanguageModels/InferenceEngines
0 commit comments