Skip to content

Commit 6265330

Browse files
author
Carina Peng
committed
Add static-shape inference engine
Static-shape LLM inference engine (StaticShapeEngine) built on the bind(into:) state binding from #156. States are discovered by name and the KV cache is right-sized per context bucket (each bucket graph is compiled with its own per-ctx strides, so a max-ctx buffer sliced down corrupts KV); ctx is parsed from the function name. Input preparation is pluggable per model family via StaticInputProvider + StaticModelProfile; StaticInputContext composes the shared InputContext and the provider mirrors SyncInputHandler. Testing: LanguageModelsTests (317) pass; correct output on gemma4-E2B, qwen2.5-1.5B, qwen3-0.6B static assets. Known limitation: cross-bucket cache re-layout needs a channel-interleave-aware copy; within-bucket generation is correct. Depends on #156.
1 parent 1677713 commit 6265330

4 files changed

Lines changed: 827 additions & 180 deletions

File tree

0 commit comments

Comments
 (0)