Support backend OpenVINO

[khanhnd61-vr]

OpenVINO backend

Runs the in-tree VLA architectures on Intel CPUs, iGPUs and NPUs through ggml's OpenVINO backend. Needs -DGGML_OPENVINO=ON; not auto-detected. Full write-up in docs/backend/ov.md.

llama.cpp is pinned at b10729. The numbers below were measured at b10331, which is byte-identical to b10729 on the CPU backend for all eleven archs, on an Intel Core Ultra X7 358H (Panther Lake) with the Arc B390 iGPU and AI Boost NPU, Ubuntu 24.04, OpenVINO 2026.2.1.

Where it stands

Seven architectures translate faithfully. One is wrong; the rest are untested or out of scope.

Arch	OpenVINO	Fidelity, CPU plugin	CPU backend	OpenVINO GPU
VLA-Adapter	Y	2.4e-6	1,228 ms	161 ms (7.6x)
Evo-1	Y	3.5e-6	3,114 ms	563 ms (5.5x)
π0.5	Y	3.5e-5	2,802 ms	683 ms (4.1x)
VLA-JEPA	Y	1.1e-4	1,046 ms	127 ms (8.2x)
GR00T N1.5	Y	6.0e-4	1,420 ms	148 ms (9.6x)
SmolVLA	Y	8.9e-4	1,364 ms	451 ms (3.0x)
GR00T N1.6	Y	1.0e-3	1,276 ms	323 ms (3.9x)
GR00T N1.7	-	1.5e0, wrong	not timed	not timed
π0, OpenVLA-OFT	~	untested, no local checkpoint	-	-
BitVLA	-	pins to CPU by design	-	-

The bar is 2.9e-3, the figure the SYCL backend is held to, measured against whichever CPU-backend reference is tighter for that arch. The iGPU plugin computes in F16 and is looser (VLA-JEPA 8.7e-3, GR00T N1.5 4.7e-3, VLA-Adapter 3.2e-3), so judge translation fidelity on the CPU plugin and treat the GPU as a separate precision target. The NPU accepts only SmolVLA and π0.5. GR00T N1.7 is not timed because it computes the wrong answer, and a latency for work that is not the same work would mislead.

What it took

Two changes in vla.cpp, both ordinary correctness fixes that are invisible on the other backends:

• Weight buffers are tagged ..._WEIGHTS instead of the default ANY, which ggml-openvino reads as "KV cache". One call site in src/loader.cpp.
• Graph tensors get unique names. ggml derives a result's name from its source, so unnamed intermediates collide, and ggml-openvino keys its translation map on those names. vla::graph_unique_names runs at each of the 29 ggml_backend_graph_compute sites and compiles to nothing off OpenVINO.

Plus one default in backend_init (GGML_OPENVINO_NAIVE_GRAPH_SIZE, so non-LLM graphs take the literal translation path), and eleven fixes to the fetched ggml OpenVINO backend, applied by scripts/patch_ggml_openvino.py at configure time. The load-bearing ones:

Fix	Assumption it breaks
GELU translated as tanh, not erf	ggml's ggml_gelu is the tanh approximation; ov's Gelu defaults to erf
Position inputs keyed per tensor	a graph has exactly one position input
Folded weights padded to full rank	a 2-D weight is only ever a GEMM operand
Naive-path compiled-model cache	(speed) that path recompiled on every graph_compute, and its graph_key is a node count plus two names, which two graphs can share
GELU_ERF translator	the op had no table entry at all

ggml-openvino is written against llama.cpp's graphs: one decoder-only transformer, one position input, an F16 KV cache. Most of these narrow an assumption stricter than the ggml contract. The position-input fix is what carries an arch through to a full prediction. The GELU fix is the highest-yield one, moving VLA-JEPA 48x and GR00T N1.5 45x, and turning both from "runs but drifts" into supported.

None of it belongs here. Every hunk is a generic ggml-openvino defect, so docs/UPSTREAMING.md splits them into eleven per-PR branches against llama.cpp master, regenerated by scripts/upstream_split.py. Landing them deletes the configure-time patch step entirely. Lead with the naive-path cache: 22.7 s to 1.4 s per prediction, still absent upstream, and it helps anyone running a graph that is not a decoder-only LLM.

Baselines

OpenVINO folds BF16 weights in as constants and executes them at F32, while ggml's CPU backend keeps them BF16. Comparing against the default reference charges the backend for a precision upgrade: it made Evo-1 look like 2.7e-3 when it is 3.5e-6. The BF16 and F32 references bracket the answer, and which one is tighter is arch-dependent (SmolVLA lands closer to BF16), so report both. For scale, the CPU backend's own output moves 2.0e-3 on SmolVLA and 1.1e-2 on VLA-JEPA from flipping that one flag.

Verification

• ctest 8/8 on CUDA, 7/7 on CPU, including a new test_graph_names that pins the renaming pass.
• All eleven archs produce an action chunk end-to-end through vla-cli on real images: ten on the CPU build, BitVLA on CUDA, which is where its int2-packed checkpoint is supported.
• vla-server drives the full protobuf and ZeroMQ path, returning a 50x32 chunk for π0.5.
• Every arch stays byte-identical to the pre-PR baseline on the CPU backend, checked with vla_predict_check before and after each commit. Nothing here changes what another backend computes.
• ASAN and UBSAN clean on the header-only tests.

Known issues

GGML_OPENVINO_CACHE_DIR returns wrong actions on a warm cache. Cold run correct, next run wrong, nothing logged. backend_init clears it and says so; VLA_ALLOW_OV_CACHE=1 keeps it if you have verified the outputs yourself.

The NPU takes two of eight archs and fails four different ways: compiler alignment rejections (Evo-1, VLA-Adapter), all-NaN output (VLA-JEPA), NPUW partitioning and plugin throws (GR00T N1.5, N1.6). None are vla.cpp's doing. It also needs two setup steps beyond the driver: libze1, and ZE_ENABLE_ALT_DRIVERS pointing at libze_intel_npu.so.1.

SmolVLA's VLA_TIMING=phase path is wrong under OpenVINO on every device. The default path that vla-server and vla-cli use is correct.

A stale build/_deps is not repatched. The patch step only runs when FetchContent repopulates, so delete it after pulling. Each hunk is now checked on its own, and the script fails loudly rather than leaving a tree that builds and is quietly wrong.

Open: GR00T N1.7

It translates and returns a full action chunk, but the values are wrong: max|delta| 1.477, rms 8.3e-2 against a peak of 0.948, 86% of 5280 values off by more than 1e-2. Deterministic, and identical on the CPU and GPU plugins.

Ruled out, each with evidence:

• Precision. Identical against BF16 and F32 references. OpenVINO is 1530x less weight-dtype-sensitive than ggml CPU here, and the model's own bf16/f32 sensitivity is rms 7.2e-4 against the error's 8.3e-2.
• Translation-path selection. The naive path is taken by default, and is_model_splitted returns false for every graph. Forcing all graphs down the decoder-only-LLM path changes the answer by 1e-5 while both stay 1.477 from the reference.
• Sequence length and token composition. Swept 3.7x (SEQ 70 to 262) by two independent routes. Relative error stayed within 0.2545 to 0.2841, and the fraction off by more than 1e-2 within 84.7% to 86.6%. Nothing accumulates.
• Matmul precision and the compiled-model cache. Both bitwise no-ops.
• The GELU mismatch that fixed VLA-JEPA and GR00T N1.5. It moves N1.7 by nothing.
• Missing or unsupported ops. N1.6 works and N1.7 does not, yet they share a 17-op vocabulary, and LM layer 0 is node-for-node identical except that N1.7's position input is [4*SEQ] for IMROPE where N1.6's is [SEQ] for NEOX. VLA-JEPA also uses IMROPE and is now clean, which weakens that lead.

Three earlier claims about N1.7 are artifacts and should not be reused:

• That its first LM block is 73-95% wrong. OpenVINO's lm_h_00..03 dumps are bit-exact copies of the input arrays and lm_h_04..15 are zeros; the main graph has exactly one output, action_pred.
• That --flash-attn 1 yields 0.948. It fails with "Got less inputs than expected" and returns no actions. 0.948 is max|reference|, what you get comparing against nothing.
• That honouring GGML_TENSOR_FLAG_OUTPUT changes the answer. Same artifact.

The failure is still unlocated, and the stage dump cannot see it because ggml-openvino writes back only true graph outputs. Next step is either a debug mode that materialises selected intermediates as ov::Results, or bisecting with cut-down graphs.

[khanhnd185]

OpenVINO backend - progress report

Status of branch backend/ov at cb78ed6 plus the working-tree changes below,
llama.cpp pinned at b10729. Reference doc: ov.md.

Measured on an Intel Core Ultra X7 358H (Panther Lake), Arc B390 iGPU, AI Boost
NPU, Ubuntu 24.04, OpenVINO 2026.2.1.

Where it stands

All nine tested architectures are inside the accuracy bar on the OpenVINO CPU
plugin, and eight of the nine on the iGPU as well.

Fidelity - compared to ggml CPU backend

Arch	CPU plugin	iGPU	NPU
Evo-1	2.2e-6	6.0e-4	compiler rejects
VLA-Adapter	3.9e-6	5.3e-3	compiler rejects
π0.5	6.1e-5	7.6e-4	1.6e-3
VLA-JEPA	7.7e-5	2.6e-3	returns NaN
GR00T N1.7	3.9e-4	2.6e-3	NPUW throws
π0	5.8e-4	6.6e-5	1.7e0 - wrong
GR00T N1.5	6.0e-4	1.6e-3	NPUW throws
GR00T N1.6	1.1e-3	1.4e-3	NPUW throws
SmolVLA	1.4e-3	2.6e-3	1.1e-2
OpenVLA-OFT	untested, no local checkpoint	-	-
BitVLA	pins to CPU by design	-	-

Latency

Model	input	CPU backend	OpenVINO CPU	OpenVINO GPU	OpenVINO NPU
VLA-JEPA	256	1,046 ms	1,265 ms	127 ms (8.2x)	returns NaN
GR00T N1.5	224	1,420 ms	2,199 ms	148 ms (9.6x)	plugin throws
VLA-Adapter	224	1,228 ms	1,603 ms	161 ms (7.6x)	not supported
GR00T N1.6	224	1,276 ms	2,256 ms	323 ms (3.9x)	plugin throws
SmolVLA	512	1,364 ms	1,340 ms	451 ms (3.0x)	1,162 ms
Evo-1	448	3,114 ms	4,523 ms	563 ms (5.5x)	not supported
π0.5	224	2,802 ms	4,285 ms	683 ms (4.1x)	916 ms
GR00T N1.7	256	1,146 ms	not timed	288 ms (4.0x)	not attempted