feat(parakeet): add Apple Core ML (Neural Engine) encoder sidecar#100
Draft
Zbig9000 wants to merge 5 commits into
Draft
feat(parakeet): add Apple Core ML (Neural Engine) encoder sidecar#100Zbig9000 wants to merge 5 commits into
Zbig9000 wants to merge 5 commits into
Conversation
Optional, Apple-only, additive acceleration for file transcription: when a
<model>-encoder.mlmodelc sits next to the GGUF on Apple hardware and the addon
is built with PARAKEET_COREML, the offline FastConformer encoder runs on the
Neural Engine via a generic MLModel wrapper; otherwise run_encoder falls back
to the ggml (Metal/CPU) encoder. Mel preprocessing, TDT/CTC decode and the
tokenizer stay on ggml. Modelled on the whisper.cpp Core ML pattern.
- src/coreml/parakeet-encoder.{h,mm}: ObjC++ sidecar with layout-adaptive I/O,
Float16/Float32 output handling and presence-driven load, folded into
libparakeet as an ARC OBJECT library.
- run_encoder: routes the full-context FastConformer through Core ML when the
sidecar is active; silent ggml fallback otherwise.
- engine: encoder_backend()/encoder_on_coreml() accessors for backend stats.
- CMake: PARAKEET_COREML option; enables OBJC+OBJCXX before add_subdirectory
(ggml) so the .mm gets a compile rule and ggml-metal .m sources stay
Objective-C under CMake 4.x.
- test-encoder-coreml-parity: per-frame cosine-similarity gate vs the ggml
encoder; skips when the sidecar is inactive.
Review StatusCurrent Status: ❌ PENDING Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member. |
…elc) Produces the offline FastConformer `<model>-encoder.mlmodelc` sidecar consumed by the PARAKEET_COREML path. Reuses the pure-PyTorch reference encoder (ref-encoder-from-gguf.py) so the exported graph matches the ggml encoder numerically, traces it at a fixed mel length (from a sample wav or an explicit count), and converts to a Core ML package (fp16 for the Neural Engine) with an optional coremlc compile step. Handles use_bias=False checkpoints (e.g. parakeet-tdt-0.6b-v3) and keeps the conv module trace-safe.
…tion parity
- Move the sidecar path derivation into its own pure TU
(src/parakeet_coreml_path.{h,cpp}) and make it quant-independent: strip a
trailing quant tag whether dot- or dash-separated (`.f16`, `.q8_0`, `-q8_0`),
so one exported encoder serves every decoder quantisation of a model
(`parakeet-tdt-0.6b-v3.q8_0.gguf` and `...f16.gguf` -> `...v3-encoder.mlmodelc`).
The previous helper only stripped the dash form, leaving the dot-form quant in
the sidecar name.
- test-coreml-paths: portable unit test for the derivation. No model, ggml or
Apple frameworks, so it runs on every CI regardless of PARAKEET_COREML.
- test-transcribe-coreml-parity: Apple-gated end-to-end gate that asserts
Engine::transcribe() yields identical text on the Core ML encoder and the ggml
encoder (each run in its own forked process). Skips when the sidecar is absent.
Validated on Apple M3 Ultra: path unit PASS, encoder cosine 1.000000, and
Core ML vs ggml transcripts identical on jfk.wav.
The Core ML wrapper previously bound only to the exact mel length the .mlmodelc was traced at: resolve_input_shape returned nil on any other length, so Engine::transcribe() silently fell back to the ggml encoder for every utterance whose length differed from the export. Since the offline encoder runs over the whole utterance in one shot (no fixed window), the sidecar effectively never engaged for arbitrary-length audio. Rebuild the input at the caller's mel length in the model's declared orientation (new shape_for_length helper) instead of requiring an exact match. A fixed-shape model at its trace length is still honoured verbatim (unchanged, including the parity gate); a fixed model at another length still fails in Core ML prediction and falls back to ggml as before; and a flexible (RangeDim / enumerated) export is driven at the requested length across utterances. This is the consumption half of variable-length support. Producing a flexible .mlmodelc is a separate exporter change: torch.jit.trace bakes the sequence length into the positional-embedding slice and attention rel_shift, so the exporter needs torch.export dynamic shapes (or scripting) plus on-device (ANE) parity validation, tracked as a follow-up.
Add an opt-in --flexible mode to the Core ML encoder exporter that declares the mel time axis as a RangeDim, so one exported .mlmodelc serves variable-length utterances instead of only the single traced length. torch.jit.trace (the fixed default) bakes the sequence length into the positional-embedding slice and attention rel_shift, so a RangeDim over that graph would still only work at the traced length. The flexible path instead: - captures via torch.export dynamic_shapes, which keeps the time axis symbolic through the graph; - swaps in a masking-free subsampling (ref.subsampling's per-stage masks are all-ones for the full-valid offline input, so this is a numerical no-op) to keep the stride-2 conv stack polymorphic in time; - bakes the relative positional table as a fixed buffer sized to the longest supported utterance and slices it by the symbolic length, leaving the dynamic slice as the only length-dependent op. The fixed default path is unchanged. Flexible export is best-effort and must be validated on Apple hardware (test-encoder-coreml-parity at several lengths + coreml-cli ANE op-placement) before use, since torch.export / coremltools dynamic-shape handling for rel_shift can need model-specific tweaks; it requires coremltools>=8 / torch>=2.3. Pairs with the wrapper change that drives a flexible encoder at the caller's length.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Optional, Apple-only, additive acceleration for Parakeet file transcription. When a
<model>-encoder.mlmodelcsits next to the GGUF on Apple hardware and the library is built with-DPARAKEET_COREML=ON, the offline FastConformer encoder runs on the Apple Neural Engine via a generic Core MLMLModelwrapper. Otherwiserun_encodersilently falls back to the ggml (Metal/CPU) encoder. Mel preprocessing, the TDT/CTC decode loop and the tokenizer always stay on ggml. Modelled on the existing whisper.cpp Core ML pattern.Selection is presence-driven (no opt-in flag): missing/corrupt/incompatible sidecar or non-Apple build -> ggml encoder. Non-Apple platforms are unaffected (feature compiles out entirely).
Changes
src/coreml/parakeet-encoder.{h,mm}: ObjC++ sidecar (ARC) with layout-adaptive tensor I/O, Float16/Float32 output handling and presence-driven load, folded intolibparakeetas an OBJECT library. The.mlmodelcexport contract is documented in the header.src/parakeet_ctc.cpp: presence-driven loader +run_encoderrouting to Core ML for the full-context encoder, with ggml fallback; backend accessors.include/parakeet/engine.h,src/parakeet_engine.cpp:encoder_backend()/encoder_on_coreml()for stats.CMakeLists.txt:PARAKEET_COREMLoption. EnablesOBJC+OBJCXXbeforeadd_subdirectory(ggml)so the.mmgets a compile rule and ggml-metal.msources stay Objective-C under CMake 4.x.test/test_encoder_coreml_parity.cpp: per-frame cosine-similarity gate vs the ggml encoder; skips when the sidecar is inactive (no-op off Apple / no sidecar).On-device validation (Apple M3 Ultra, macOS 15.5, Xcode 16.4, CMake 4.1.1)
-DPARAKEET_COREML=ON(Metal + Core ML together):libparakeet.dyliblinks;parakeet-encoder.mmcompiles as ObjC++/ARC against the real CoreML/Foundation SDK.test-encoder-coreml-paritycompiles and links on Apple..mlmodelcmatching the export contract, driving the real built wrapper object: initbackend=coreml, encoderc=0, exact round-trip in FP32 (cos=1.0) and within tolerance in FP16 (cos=0.99999998, exercising thedecode_float16path).Not covered here (follow-ups, need real assets)
.mlmodelc+coreml-cliANE op-placement verification.run_encoderrouting end-to-end) and numerical parity vs ggml.Related
coremlfeature for Darwin/iOS prebuilds (qvac-registry-vcpkg).encoderBackend/encoderOnCoreml) in@qvac/transcription-parakeet.