Notes for building vla.cpp against ggml's OpenVINO backend, and an honest
account of how far it currently runs. Like SYCL, OpenVINO is not
auto-detected: it needs an explicit -DGGML_OPENVINO=ON and the OpenVINO
runtime on the configure line.
Status: every architecture that can reach this backend translates faithfully, on the CPU plugin and on the iGPU. That is ten of the eleven in the tree - SmolVLA, π0, π0.5, Evo-1, VLA-Adapter, GR00T N1.5, GR00T N1.6, GR00T N1.7, VLA-JEPA and OpenVLA-OFT - all agreeing with a CPU-backend reference to 1.4e-3 or better on the OpenVINO CPU plugin, and nine of the ten are inside the same bar on the Arc B390 iGPU, where the speedup over the native CPU backend runs from 3.0x to 9.6x. The eleventh, BitVLA, pins its ggml graph to the CPU backend by design and never reaches this backend at all.
Fifteen fixes were needed, thirteen of them inside ggml's OpenVINO backend, which is written against llama.cpp's graphs and had never seen a vision tower or an action expert - see What had to change.
Note the baseline: OpenVINO executes the checkpoint's BF16 weights at F32, so compare against
--weight-dtype f32or you will charge the backend for a precision upgrade. See Picking the right baseline.
Measured on an Intel Core Ultra X7 358H (Panther Lake) with the Arc B390
iGPU and the AI Boost NPU, Ubuntu 24.04, OpenVINO 2026.2.1, llama.cpp b10729,
on the checkpoints under vrfai/ on the Hub. Every fidelity number was
re-measured on that pin; only the latency table still dates from b10331.
ggml's backend translates a ggml compute graph into an OpenVINO model and hands
it to the CPU, GPU or NPU plugin, which compiles and fuses it for the device.
Unlike SYCL it needs no separate compiler: the stock GCC/Clang build links
libopenvino.
Intel CPUs, Intel GPUs (integrated Xe / Arc, and discrete), and Intel NPUs (Core Ultra). Linux only here - Ubuntu 22.04 or 24.04.
The CPU plugin needs nothing. The GPU and NPU plugins reach the hardware through
/dev/dri/renderD* and /dev/accel/accel0, both owned by the render group:
sudo usermod -aG render,video "$USER" # re-login afterwards
clinfo -l # must enumerate the GPUNumber of platforms 0 with /etc/OpenCL/vendors/intel.icd present almost
always means the render group has not taken effect yet. Without it,
GGML_OPENVINO_DEVICE=GPU warns and silently falls back to the CPU plugin.
For the GPU compute runtime and NPU driver packages themselves, follow llama.cpp's OpenVINO notes.
The NPU needs two more things its driver packages do not pull in. Neither failure
is reported as an error - the device simply does not appear and every
GGML_OPENVINO_DEVICE=NPU run lands on the CPU plugin instead
(device NPU is not available, fallback to CPU).
sudo apt-get install -y libze1 # the Level Zero loader; the driver alone is not enough
# check with: ldconfig -p | grep ze_loader
export ZE_ENABLE_ALT_DRIVERS=/lib/x86_64-linux-gnu/libze_intel_npu.so.1Ubuntu's loader (1.16.1 in noble) does not discover libze_intel_npu.so.1 on its
own, hence the override; a loader from Intel's own graphics repository,
version-matched to the driver, should not need it - untested here. With both in
place the device enumerates as NPU Intel(R) AI Boost.
sudo apt-get install -y opencl-clhpp-headers ocl-icd-opencl-dev opencl-headers \
cmake ninja-build pkg-config protobuf-compiler libprotobuf-dev \
libzmq3-dev cppzmq-devThen either install OpenVINO
from the archive
by hand, or run bash scripts/install_ov.sh, which also pulls the GPU driver
stack and adds you to render.
source /opt/intel/openvino/setupvars.sh
cmake -B build-ov -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON
cmake --build build-ov -j$(nproc)setupvars.sh must be sourced in every shell that builds or runs the binaries:
libopenvino.so and its TBB live under /opt/intel. Configure fails early with
a pointer back here if the runtime is not on CMAKE_PREFIX_PATH.
scripts/patch_ggml_openvino.py runs as the FetchContent patch step, so the
thirteen ggml fixes are applied automatically - there is no manual git apply.
The step only runs when FetchContent populates the source dir, so a build/_deps
left over from an older checkout keeps the hunks it was patched with: delete it
after pulling rather than trusting a reconfigure. The script checks each hunk on
its own and fails loudly on a tree it cannot bring up to date.
GGML_OPENVINO_DEVICE picks the target by name (VLA_DEVICE does not apply -
ggml exposes OpenVINO as a single device):
GGML_OPENVINO_DEVICE=GPU ./build-ov/vla-server ./weights/smolvla-libero.ggufTwo lines identify the selection at startup:
OpenVINO: using device GPU
vla: backend = OPENVINO (asked for GPU, see ggml's "using device" line)
The first comes from ggml and is authoritative: an unavailable device logs a
warning there and falls back to CPU, which is still the OpenVINO CPU plugin,
not ggml's native CPU backend. The second echoes what was requested, so the pair
tells you whether you got the device you asked for.
OpenVINO compiles each graph on first use, which is slow - a minute or two for a
vision tower on the GPU. Compiled graphs are then cached in-process for the life
of the model, so only the first prediction pays that; give any client a receive
timeout well above the first request. Do not set GGML_OPENVINO_CACHE_DIR to
carry them across restarts; it produces silently wrong actions here - see
Known issues.
vla_predict_check (a test target - add -DVLA_BUILD_TESTS=ON), fixed noise,
one camera view, best of 4-6 iterations after 3 warmups. "CPU backend" is ggml's
own CPU backend on the same 16-core host. No GGML_OPENVINO_CACHE_DIR.
Latencies were taken at b10331 and have not been re-timed on b10729. Read the
GPU column for VLA-JEPA and GR00T N1.7 as the cost of a wrong answer at that pin;
both are correct now.
| Model | input | CPU backend | OpenVINO CPU | OpenVINO GPU | OpenVINO NPU |
|---|---|---|---|---|---|
| VLA-JEPA | 256 | 1,046 ms | 1,265 ms | 127 ms (8.2x) | returns NaN |
| GR00T N1.5 | 224 | 1,420 ms | 2,199 ms | 148 ms (9.6x) | plugin throws |
| VLA-Adapter | 224 | 1,228 ms | 1,603 ms | 161 ms (7.6x) | not supported |
| GR00T N1.6 | 224 | 1,276 ms | 2,256 ms | 323 ms (3.9x) | plugin throws |
| SmolVLA | 512 | 1,364 ms | 1,340 ms | 451 ms (3.0x) | 1,162 ms |
| Evo-1 | 448 | 3,114 ms | 4,523 ms | 563 ms (5.5x) | not supported |
| π0.5 | 224 | 2,802 ms | 4,285 ms | 683 ms (4.1x) | 916 ms |
| GR00T N1.7 | 256 | 1,146 ms | not timed | 288 ms (4.0x) | not attempted |
The iGPU is the reason to use this backend, and it pays off most where the model is most vision-heavy. The OpenVINO CPU plugin is at best parity with ggml's own CPU backend and often well behind it, so it is only worth running to debug a translation. The NPU beats the CPU backend on the two models it accepts while drawing far less power, which is the interesting result for a robot - see the NPU limits under Known issues.
Actions are checked against the CPU backend on identical inputs; both sides are
deterministic, so the numbers are exact rather than sampled. But which CPU run
you compare against matters. OpenVINO folds the checkpoint's BF16 weights in as
constants and its CPU plugin executes them at F32, while ggml's CPU backend keeps
them BF16, so a naive comparison charges the OpenVINO backend for a precision
upgrade. Running the reference with --weight-dtype f32 removes that term.
The two references bracket the answer, and which is tighter turns on how much of a given checkpoint is BF16 in the first place. Report both and take the smaller:
| Model | vs BF16 reference | vs F32 reference | tighter reference |
|---|---|---|---|
| VLA-Adapter | 2.1e-3 | 2.4e-6 | F32 |
| Evo-1 | 2.7e-3 | 3.5e-6 | F32 |
| π0.5 | 7.7e-4 | 3.5e-5 | F32 |
| VLA-JEPA | 1.1e-2 | 1.1e-4 | F32 |
| GR00T N1.5 | 5.5e-3 | 6.0e-4 | F32 |
| GR00T N1.6 | 5.0e-3 | 1.0e-3 | F32 |
| SmolVLA | 8.9e-4 | 1.3e-3 | BF16 |
| GR00T N1.7 | 1.5e0 | 4.2e-4 | F32 |
Evo-1 and VLA-Adapter agree with an F32 reference to six decimal places, which is as close to "the translation is exact" as this harness can show; the BF16 comparison for those two was measuring nothing but the dtype. SmolVLA is the counterexample that stops this being a universal rule. For context, the CPU backend's own output moves by 2.0e-3 (SmolVLA), 2.7e-3 (Evo-1) or 1.1e-2 (VLA-JEPA) when you flip that one flag, so the model's intrinsic sensitivity to precision is the same size as the numbers being reported.
Against the F32 reference, which is the fidelity number:
| Model | OpenVINO CPU | OpenVINO GPU | OpenVINO NPU |
|---|---|---|---|
| Evo-1 | 2.2e-6 | 6.0e-4 | compiler rejects |
| VLA-Adapter | 3.9e-6 | 6.8e-3 | compiler rejects |
| OpenVLA-OFT | 3.9e-6 | 2.2e-3 | compiler rejects |
| π0.5 | 6.1e-5 | 7.6e-4 | 1.6e-3 |
| VLA-JEPA | 7.7e-5 | 2.6e-3 | returns NaN |
| GR00T N1.7 | 3.9e-4 | 2.6e-3 | NPUW throws |
| π0 | 5.8e-4 | 6.6e-5 | 1.7e0 |
| GR00T N1.5 | 6.0e-4 | 4.6e-3 | NPUW throws |
| GR00T N1.6 | 1.1e-3 | 1.4e-3 | NPUW throws |
| SmolVLA | 1.4e-3 | 2.6e-3 | 1.1e-2 |
Every tested arch is inside the 2.9e-3 bar on the CPU plugin, most by one to three orders of magnitude, and nine of the ten are inside it on the GPU as well (GR00T N1.5's 4.6e-3 against F32 is 1.6e-3 against BF16, the tighter reference for that arch). The one outside is VLA-Adapter, at 6.8e-3 against F32 on actions peaking at 0.62 - the GPU plugin computing in F16, a precision effect rather than a translation error, since the same arch is 3.9e-6 on the CPU plugin. Judge translation fidelity on the CPU plugin and treat the GPU as a separate precision target. π0 is the exception in the other direction, tighter on the GPU (6.6e-5) because it is the one arch that runs the GPU at F32; see Known issues, which also covers the NPU column.
Fifteen fixes: two in vla.cpp, thirteen in ggml's OpenVINO backend. Both
vla.cpp-side ones are ordinary correctness fixes that happen to be invisible on
the other backends - vla_predict_check on a CPU build of this branch is
byte-identical to the same build of the base commit, for every model tested.
- Weight buffers are tagged.
ggml_backend_alloc_ctx_tensorsleaves a buffer onGGML_BACKEND_BUFFER_USAGE_ANY, and ggml-openvino reads ANY as "KV cache", giving every weight a dynamic sequence dimension.vla::alloc_weightsinsrc/backend.htags it..._WEIGHTS, which is what lets the frontend fold weights in as constants. Since 0.3.0 every arch allocates throughvla::WeightLoader, so this is one call site insrc/loader.cpp. - Graph tensors get unique names. ggml derives a result's name from its
source, so a graph whose intermediates were never named ends up with many
tensors sharing one name (
" (reshaped)"and friends). ggml-openvino keys its translation map on those names, so duplicates silently collapse into one node and the graph wires the wrong tensor into the next op.vla::graph_unique_namesrelabels duplicates before compute, at each of the 29ggml_backend_graph_computecall sites. It compiles to nothing outside an OpenVINO build.
backend_init also sets one default, the way the SYCL rung already sets
GGML_SYCL_ENABLE_VMM=0: GGML_OPENVINO_NAIVE_GRAPH_SIZE defaults high.
ggml-openvino translates a graph under 20 nodes literally and sends anything
larger through a model builder that assumes a decoder-only LLM. The literal path
is the one that fits a vision tower and an action expert. An explicit setting
still wins.
The other thirteen are applied to ggml's OpenVINO backend by
scripts/patch_ggml_openvino.py at configure time; its docstring carries the
per-fix detail. Each narrows an llama.cpp-shaped assumption that is stricter than
the ggml contract, or fills a gap:
| Fix | What it addresses |
|---|---|
| PERMUTE op_case 2 requires a ROPE | assumes any permute of a view is a rope'd query |
| Two elementwise adds never stacked on a GEMM | the GPU plugin folds both in as post-ops and drops the second operand |
| GPU inference precision exposed | the plugin's F16 default compounds through a denoise loop unrolled in one graph |
| GELU translated as tanh, not erf | assumes ggml's GELU is the exact erf form |
| Intel OpenCL platform selection | assumes the first OpenCL platform is Intel's |
RESHAPE op_case guard |
assumes a reshape flattening dims 0-2 is the KV-cache flatten |
| SDPA K/V converted with Q | assumes K/V arrive as F16 because the KV cache is |
| Position inputs keyed per tensor | assumes a graph has exactly one position input |
| Folded weights padded to full rank | a 2-D weight becomes a rank-2 constant, but views index it at ggml rank |
| CONCAT input ranks aligned | same rank-2 constants, and concat cannot broadcast rank |
Missing GELU_ERF translator |
the exact-erf GELU op had no table entry at all, so a graph using it could not run |
| Naive-path graph cache | that path re-compiled the whole model on every graph_compute, and its graph_key is a node count plus two names, which two graphs can share |
| Interleaved-mrope sectors bounded | the sector cycle ignored sections, so the last few took the wrong stream |
| Naive-path threshold settable | the 20-node constant is what picks the literal path |
The bolded rows are the ones that turned a wrong arch into a correct one: PERMUTE op_case 2 for GR00T N1.7, the double-elementwise guard for the Eagle-VLM archs on the iGPU, the GELU mode for VLA-JEPA and GR00T N1.5, per-tensor position inputs for SmolVLA (which passes three position tensors, so they aliased). The GELU mode and the position-input fix are the two worth upstreaming.
Debugging a mistranslation. The backend writes back true graph outputs and
nothing else, so an interior tensor is observable only two ways: truncate the
graph at a stage, which makes that stage the terminal node, or set
GGML_OPENVINO_DEBUG_NODE to materialise one node as an extra ov::Result. Both
of the subtlest fixes above were found by bisecting that way, comparing each
stage against a CPU-backend reference; neither logs anything when it goes wrong.
π0 needs F32 on the GPU, and gets it by default. The GPU plugin computes in
F16, which is most of why it is fast. π0 unrolls its whole 10-step denoise loop
inside a single graph, so that error compounds with nothing to reset it: its
continuous action dims land 4e-2 from an F32 reference and its gripper - a
saturating ±1 channel - crosses its threshold one step late. That reads as
max|delta| 1.7; on a robot it is a late grasp. GGML_OPENVINO_GPU_PRECISION=f32
puts it back at 6.5e-5, and backend_init defaults it for π0 alone because it
costs about 3x (383 ms -> 1,170 ms). Set =f16 to override. No other arch needs
it.
Do not set GGML_OPENVINO_CACHE_DIR. OpenVINO's on-disk blob cache reloads a
compiled graph that computes the wrong thing. A cold run against a fresh cache
directory is correct; the very next run, reading back the blobs it just wrote, is
not - max|delta| 1.2e-3 cold, 2.9e0 warm, with nothing logged, which for a policy
server is the worst possible failure mode. backend_init therefore clears the
variable and says so; VLA_ALLOW_OV_CACHE=1 keeps it. Unverified guess at the
cause: the blob key does not capture something that differs between vla.cpp's
several graphs, so one graph gets another's blob - the same class of bug as the
in-process graph_key above, which is now keyed on shapes. In practice, pay the
compile once per process and leave it unset.
The NPU accepts three of the ten archs, and only two of those are correct. Check every NPU run against the startup banner: an unavailable NPU falls back to the CPU plugin silently and would otherwise report excellent numbers that are not NPU numbers at all. A partially-failing run still reports a wall-clock time, so do not read a latency off a run whose actions did not come out.
| Model | NPU outcome |
|---|---|
| SmolVLA | runs, 1.1e-2 (the compile config's dynamic quantization) |
| π0.5 | runs, 1.6e-3 |
| π0 | runs, but 1.7e0 wrong - see below |
| VLA-JEPA | compiles and runs, returns all NaN (not diagnosed) |
| GR00T N1.5 / N1.6 / N1.7 | NPUW: Assertion all_ok failed, partitioning.cpp:1350 |
| Evo-1 | compiler rejects: Input channels '1025' is not aligned by '16' |
| VLA-Adapter | compiler rejects: Input channels '261' is not aligned by '16' |
| OpenVLA-OFT | compiler rejects: Input channels '261' is not aligned by '16' |
The three alignment rejections are Intel's NPU compiler: 1025 is Evo-1's 1024 patches plus a CLS token, and 261 is 256 plus 5 - VLA-Adapter and OpenVLA-OFT hit the identical number for the identical reason. SmolVLA and π0.5 happen to have 16-aligned sequence lengths. None of these are vla.cpp's doing.
π0 on the NPU is the same bug as π0 on the GPU, and here there is no remedy.
Continuous dims 0-5 land at 4.0e-2 and the gripper flips at step 44, exactly as
on the GPU. But the GPU fix does not transfer: setting the inference precision to
F32 makes the NPU refuse to compile at all (core.cpp:117), because the hint
conflicts with the NPUW and dynamic-quantization config the NPU path sets up. So
GGML_OPENVINO_GPU_PRECISION is GPU-only by necessity, and π0 should not be run
on the NPU.
SmolVLA's VLA_TIMING=phase path is wrong under OpenVINO. SmolVLA has a
second graph builder used when a caller asks for per-phase timings, and it does
not survive translation - max|delta| 1.9 on every device. The default
TimingDetail::NONE path, which is what vla-server and vla-cli use, is
correct, and on the native CPU backend the two paths agree exactly. Evo-1 and
π0.5 are unaffected on the same path, so this is specific to SmolVLA's second
graph; one hypothesis - that the split-graph guard sends it down the LLM path -
was tested and is wrong. Per-stage timings for SmolVLA are omitted from the
tables above.
Fixing issue channel not aligned by 16. Feasible solution is padding dummy channel so that number of channels is a multiple of 16.
Splitting across devices. Intel's own π0.5 write-up puts the vision encoder and language model on the iGPU and the action expert on the NPU, with the KV cache as the only cross-device handoff. The toolchain does not carry over - PyTorch exported to OpenVINO IR as three separate models, no ggml - but the shape of the answer does: the two devices suit different stages, and π0.5 on the NPU alone is already within 1.5x of the iGPU at a fraction of the power. vla.cpp cannot make that split today because the core drives one backend for a whole prediction. It would need a per-stage backend rather than a per-op scheduler - the vision tower, the prefix and the action expert already hand off through host memory, so the seam is in the right place - but that is an engine change, not a backend one.