Skip to content

Latest commit

 

History

History
367 lines (304 loc) · 19.6 KB

File metadata and controls

367 lines (304 loc) · 19.6 KB

vla.cpp on Intel CPUs, GPUs and NPUs (OpenVINO backend)

Notes for building vla.cpp against ggml's OpenVINO backend, and an honest account of how far it currently runs. Like SYCL, OpenVINO is not auto-detected: it needs an explicit -DGGML_OPENVINO=ON and the OpenVINO runtime on the configure line.

Status: every architecture that can reach this backend translates faithfully, on the CPU plugin and on the iGPU. That is ten of the eleven in the tree - SmolVLA, π0, π0.5, Evo-1, VLA-Adapter, GR00T N1.5, GR00T N1.6, GR00T N1.7, VLA-JEPA and OpenVLA-OFT - all agreeing with a CPU-backend reference to 1.4e-3 or better on the OpenVINO CPU plugin, and nine of the ten are inside the same bar on the Arc B390 iGPU, where the speedup over the native CPU backend runs from 3.0x to 9.6x. The eleventh, BitVLA, pins its ggml graph to the CPU backend by design and never reaches this backend at all.

Fifteen fixes were needed, thirteen of them inside ggml's OpenVINO backend, which is written against llama.cpp's graphs and had never seen a vision tower or an action expert - see What had to change.

Note the baseline: OpenVINO executes the checkpoint's BF16 weights at F32, so compare against --weight-dtype f32 or you will charge the backend for a precision upgrade. See Picking the right baseline.

Measured on an Intel Core Ultra X7 358H (Panther Lake) with the Arc B390 iGPU and the AI Boost NPU, Ubuntu 24.04, OpenVINO 2026.2.1, llama.cpp b10729, on the checkpoints under vrfai/ on the Hub. Every fidelity number was re-measured on that pin; only the latency table still dates from b10331.

ggml's backend translates a ggml compute graph into an OpenVINO model and hands it to the CPU, GPU or NPU plugin, which compiles and fuses it for the device. Unlike SYCL it needs no separate compiler: the stock GCC/Clang build links libopenvino.

Supported devices

Intel CPUs, Intel GPUs (integrated Xe / Arc, and discrete), and Intel NPUs (Core Ultra). Linux only here - Ubuntu 22.04 or 24.04.

Prerequisites

1. Device access

The CPU plugin needs nothing. The GPU and NPU plugins reach the hardware through /dev/dri/renderD* and /dev/accel/accel0, both owned by the render group:

sudo usermod -aG render,video "$USER"   # re-login afterwards
clinfo -l                               # must enumerate the GPU

Number of platforms 0 with /etc/OpenCL/vendors/intel.icd present almost always means the render group has not taken effect yet. Without it, GGML_OPENVINO_DEVICE=GPU warns and silently falls back to the CPU plugin.

For the GPU compute runtime and NPU driver packages themselves, follow llama.cpp's OpenVINO notes.

The NPU needs two more things its driver packages do not pull in. Neither failure is reported as an error - the device simply does not appear and every GGML_OPENVINO_DEVICE=NPU run lands on the CPU plugin instead (device NPU is not available, fallback to CPU).

sudo apt-get install -y libze1   # the Level Zero loader; the driver alone is not enough
                                 # check with: ldconfig -p | grep ze_loader
export ZE_ENABLE_ALT_DRIVERS=/lib/x86_64-linux-gnu/libze_intel_npu.so.1

Ubuntu's loader (1.16.1 in noble) does not discover libze_intel_npu.so.1 on its own, hence the override; a loader from Intel's own graphics repository, version-matched to the driver, should not need it - untested here. With both in place the device enumerates as NPU Intel(R) AI Boost.

2. OpenVINO runtime + OpenCL headers

sudo apt-get install -y opencl-clhpp-headers ocl-icd-opencl-dev opencl-headers \
    cmake ninja-build pkg-config protobuf-compiler libprotobuf-dev \
    libzmq3-dev cppzmq-dev

Then either install OpenVINO from the archive by hand, or run bash scripts/install_ov.sh, which also pulls the GPU driver stack and adds you to render.

Configure & build

source /opt/intel/openvino/setupvars.sh

cmake -B build-ov -G Ninja -DCMAKE_BUILD_TYPE=Release -DGGML_OPENVINO=ON
cmake --build build-ov -j$(nproc)

setupvars.sh must be sourced in every shell that builds or runs the binaries: libopenvino.so and its TBB live under /opt/intel. Configure fails early with a pointer back here if the runtime is not on CMAKE_PREFIX_PATH.

scripts/patch_ggml_openvino.py runs as the FetchContent patch step, so the thirteen ggml fixes are applied automatically - there is no manual git apply. The step only runs when FetchContent populates the source dir, so a build/_deps left over from an older checkout keeps the hunks it was patched with: delete it after pulling rather than trusting a reconfigure. The script checks each hunk on its own and fails loudly on a tree it cannot bring up to date.

Run

GGML_OPENVINO_DEVICE picks the target by name (VLA_DEVICE does not apply - ggml exposes OpenVINO as a single device):

GGML_OPENVINO_DEVICE=GPU ./build-ov/vla-server ./weights/smolvla-libero.gguf

Two lines identify the selection at startup:

OpenVINO: using device GPU
vla: backend = OPENVINO (asked for GPU, see ggml's "using device" line)

The first comes from ggml and is authoritative: an unavailable device logs a warning there and falls back to CPU, which is still the OpenVINO CPU plugin, not ggml's native CPU backend. The second echoes what was requested, so the pair tells you whether you got the device you asked for.

OpenVINO compiles each graph on first use, which is slow - a minute or two for a vision tower on the GPU. Compiled graphs are then cached in-process for the life of the model, so only the first prediction pays that; give any client a receive timeout well above the first request. Do not set GGML_OPENVINO_CACHE_DIR to carry them across restarts; it produces silently wrong actions here - see Known issues.

Results

vla_predict_check (a test target - add -DVLA_BUILD_TESTS=ON), fixed noise, one camera view, best of 4-6 iterations after 3 warmups. "CPU backend" is ggml's own CPU backend on the same 16-core host. No GGML_OPENVINO_CACHE_DIR.

Latencies were taken at b10331 and have not been re-timed on b10729. Read the GPU column for VLA-JEPA and GR00T N1.7 as the cost of a wrong answer at that pin; both are correct now.

Model input CPU backend OpenVINO CPU OpenVINO GPU OpenVINO NPU
VLA-JEPA 256 1,046 ms 1,265 ms 127 ms (8.2x) returns NaN
GR00T N1.5 224 1,420 ms 2,199 ms 148 ms (9.6x) plugin throws
VLA-Adapter 224 1,228 ms 1,603 ms 161 ms (7.6x) not supported
GR00T N1.6 224 1,276 ms 2,256 ms 323 ms (3.9x) plugin throws
SmolVLA 512 1,364 ms 1,340 ms 451 ms (3.0x) 1,162 ms
Evo-1 448 3,114 ms 4,523 ms 563 ms (5.5x) not supported
π0.5 224 2,802 ms 4,285 ms 683 ms (4.1x) 916 ms
GR00T N1.7 256 1,146 ms not timed 288 ms (4.0x) not attempted

The iGPU is the reason to use this backend, and it pays off most where the model is most vision-heavy. The OpenVINO CPU plugin is at best parity with ggml's own CPU backend and often well behind it, so it is only worth running to debug a translation. The NPU beats the CPU backend on the two models it accepts while drawing far less power, which is the interesting result for a robot - see the NPU limits under Known issues.

Picking the right baseline

Actions are checked against the CPU backend on identical inputs; both sides are deterministic, so the numbers are exact rather than sampled. But which CPU run you compare against matters. OpenVINO folds the checkpoint's BF16 weights in as constants and its CPU plugin executes them at F32, while ggml's CPU backend keeps them BF16, so a naive comparison charges the OpenVINO backend for a precision upgrade. Running the reference with --weight-dtype f32 removes that term.

The two references bracket the answer, and which is tighter turns on how much of a given checkpoint is BF16 in the first place. Report both and take the smaller:

Model vs BF16 reference vs F32 reference tighter reference
VLA-Adapter 2.1e-3 2.4e-6 F32
Evo-1 2.7e-3 3.5e-6 F32
π0.5 7.7e-4 3.5e-5 F32
VLA-JEPA 1.1e-2 1.1e-4 F32
GR00T N1.5 5.5e-3 6.0e-4 F32
GR00T N1.6 5.0e-3 1.0e-3 F32
SmolVLA 8.9e-4 1.3e-3 BF16
GR00T N1.7 1.5e0 4.2e-4 F32

Evo-1 and VLA-Adapter agree with an F32 reference to six decimal places, which is as close to "the translation is exact" as this harness can show; the BF16 comparison for those two was measuring nothing but the dtype. SmolVLA is the counterexample that stops this being a universal rule. For context, the CPU backend's own output moves by 2.0e-3 (SmolVLA), 2.7e-3 (Evo-1) or 1.1e-2 (VLA-JEPA) when you flip that one flag, so the model's intrinsic sensitivity to precision is the same size as the numbers being reported.

Full results

Against the F32 reference, which is the fidelity number:

Model OpenVINO CPU OpenVINO GPU OpenVINO NPU
Evo-1 2.2e-6 6.0e-4 compiler rejects
VLA-Adapter 3.9e-6 6.8e-3 compiler rejects
OpenVLA-OFT 3.9e-6 2.2e-3 compiler rejects
π0.5 6.1e-5 7.6e-4 1.6e-3
VLA-JEPA 7.7e-5 2.6e-3 returns NaN
GR00T N1.7 3.9e-4 2.6e-3 NPUW throws
π0 5.8e-4 6.6e-5 1.7e0
GR00T N1.5 6.0e-4 4.6e-3 NPUW throws
GR00T N1.6 1.1e-3 1.4e-3 NPUW throws
SmolVLA 1.4e-3 2.6e-3 1.1e-2

Every tested arch is inside the 2.9e-3 bar on the CPU plugin, most by one to three orders of magnitude, and nine of the ten are inside it on the GPU as well (GR00T N1.5's 4.6e-3 against F32 is 1.6e-3 against BF16, the tighter reference for that arch). The one outside is VLA-Adapter, at 6.8e-3 against F32 on actions peaking at 0.62 - the GPU plugin computing in F16, a precision effect rather than a translation error, since the same arch is 3.9e-6 on the CPU plugin. Judge translation fidelity on the CPU plugin and treat the GPU as a separate precision target. π0 is the exception in the other direction, tighter on the GPU (6.6e-5) because it is the one arch that runs the GPU at F32; see Known issues, which also covers the NPU column.

What had to change

Fifteen fixes: two in vla.cpp, thirteen in ggml's OpenVINO backend. Both vla.cpp-side ones are ordinary correctness fixes that happen to be invisible on the other backends - vla_predict_check on a CPU build of this branch is byte-identical to the same build of the base commit, for every model tested.

  • Weight buffers are tagged. ggml_backend_alloc_ctx_tensors leaves a buffer on GGML_BACKEND_BUFFER_USAGE_ANY, and ggml-openvino reads ANY as "KV cache", giving every weight a dynamic sequence dimension. vla::alloc_weights in src/backend.h tags it ..._WEIGHTS, which is what lets the frontend fold weights in as constants. Since 0.3.0 every arch allocates through vla::WeightLoader, so this is one call site in src/loader.cpp.
  • Graph tensors get unique names. ggml derives a result's name from its source, so a graph whose intermediates were never named ends up with many tensors sharing one name (" (reshaped)" and friends). ggml-openvino keys its translation map on those names, so duplicates silently collapse into one node and the graph wires the wrong tensor into the next op. vla::graph_unique_names relabels duplicates before compute, at each of the 29 ggml_backend_graph_compute call sites. It compiles to nothing outside an OpenVINO build.

backend_init also sets one default, the way the SYCL rung already sets GGML_SYCL_ENABLE_VMM=0: GGML_OPENVINO_NAIVE_GRAPH_SIZE defaults high. ggml-openvino translates a graph under 20 nodes literally and sends anything larger through a model builder that assumes a decoder-only LLM. The literal path is the one that fits a vision tower and an action expert. An explicit setting still wins.

The other thirteen are applied to ggml's OpenVINO backend by scripts/patch_ggml_openvino.py at configure time; its docstring carries the per-fix detail. Each narrows an llama.cpp-shaped assumption that is stricter than the ggml contract, or fills a gap:

Fix What it addresses
PERMUTE op_case 2 requires a ROPE assumes any permute of a view is a rope'd query
Two elementwise adds never stacked on a GEMM the GPU plugin folds both in as post-ops and drops the second operand
GPU inference precision exposed the plugin's F16 default compounds through a denoise loop unrolled in one graph
GELU translated as tanh, not erf assumes ggml's GELU is the exact erf form
Intel OpenCL platform selection assumes the first OpenCL platform is Intel's
RESHAPE op_case guard assumes a reshape flattening dims 0-2 is the KV-cache flatten
SDPA K/V converted with Q assumes K/V arrive as F16 because the KV cache is
Position inputs keyed per tensor assumes a graph has exactly one position input
Folded weights padded to full rank a 2-D weight becomes a rank-2 constant, but views index it at ggml rank
CONCAT input ranks aligned same rank-2 constants, and concat cannot broadcast rank
Missing GELU_ERF translator the exact-erf GELU op had no table entry at all, so a graph using it could not run
Naive-path graph cache that path re-compiled the whole model on every graph_compute, and its graph_key is a node count plus two names, which two graphs can share
Interleaved-mrope sectors bounded the sector cycle ignored sections, so the last few took the wrong stream
Naive-path threshold settable the 20-node constant is what picks the literal path

The bolded rows are the ones that turned a wrong arch into a correct one: PERMUTE op_case 2 for GR00T N1.7, the double-elementwise guard for the Eagle-VLM archs on the iGPU, the GELU mode for VLA-JEPA and GR00T N1.5, per-tensor position inputs for SmolVLA (which passes three position tensors, so they aliased). The GELU mode and the position-input fix are the two worth upstreaming.

Debugging a mistranslation. The backend writes back true graph outputs and nothing else, so an interior tensor is observable only two ways: truncate the graph at a stage, which makes that stage the terminal node, or set GGML_OPENVINO_DEBUG_NODE to materialise one node as an extra ov::Result. Both of the subtlest fixes above were found by bisecting that way, comparing each stage against a CPU-backend reference; neither logs anything when it goes wrong.

Known issues

π0 needs F32 on the GPU, and gets it by default. The GPU plugin computes in F16, which is most of why it is fast. π0 unrolls its whole 10-step denoise loop inside a single graph, so that error compounds with nothing to reset it: its continuous action dims land 4e-2 from an F32 reference and its gripper - a saturating ±1 channel - crosses its threshold one step late. That reads as max|delta| 1.7; on a robot it is a late grasp. GGML_OPENVINO_GPU_PRECISION=f32 puts it back at 6.5e-5, and backend_init defaults it for π0 alone because it costs about 3x (383 ms -> 1,170 ms). Set =f16 to override. No other arch needs it.

Do not set GGML_OPENVINO_CACHE_DIR. OpenVINO's on-disk blob cache reloads a compiled graph that computes the wrong thing. A cold run against a fresh cache directory is correct; the very next run, reading back the blobs it just wrote, is not - max|delta| 1.2e-3 cold, 2.9e0 warm, with nothing logged, which for a policy server is the worst possible failure mode. backend_init therefore clears the variable and says so; VLA_ALLOW_OV_CACHE=1 keeps it. Unverified guess at the cause: the blob key does not capture something that differs between vla.cpp's several graphs, so one graph gets another's blob - the same class of bug as the in-process graph_key above, which is now keyed on shapes. In practice, pay the compile once per process and leave it unset.

The NPU accepts three of the ten archs, and only two of those are correct. Check every NPU run against the startup banner: an unavailable NPU falls back to the CPU plugin silently and would otherwise report excellent numbers that are not NPU numbers at all. A partially-failing run still reports a wall-clock time, so do not read a latency off a run whose actions did not come out.

Model NPU outcome
SmolVLA runs, 1.1e-2 (the compile config's dynamic quantization)
π0.5 runs, 1.6e-3
π0 runs, but 1.7e0 wrong - see below
VLA-JEPA compiles and runs, returns all NaN (not diagnosed)
GR00T N1.5 / N1.6 / N1.7 NPUW: Assertion all_ok failed, partitioning.cpp:1350
Evo-1 compiler rejects: Input channels '1025' is not aligned by '16'
VLA-Adapter compiler rejects: Input channels '261' is not aligned by '16'
OpenVLA-OFT compiler rejects: Input channels '261' is not aligned by '16'

The three alignment rejections are Intel's NPU compiler: 1025 is Evo-1's 1024 patches plus a CLS token, and 261 is 256 plus 5 - VLA-Adapter and OpenVLA-OFT hit the identical number for the identical reason. SmolVLA and π0.5 happen to have 16-aligned sequence lengths. None of these are vla.cpp's doing.

π0 on the NPU is the same bug as π0 on the GPU, and here there is no remedy. Continuous dims 0-5 land at 4.0e-2 and the gripper flips at step 44, exactly as on the GPU. But the GPU fix does not transfer: setting the inference precision to F32 makes the NPU refuse to compile at all (core.cpp:117), because the hint conflicts with the NPUW and dynamic-quantization config the NPU path sets up. So GGML_OPENVINO_GPU_PRECISION is GPU-only by necessity, and π0 should not be run on the NPU.

SmolVLA's VLA_TIMING=phase path is wrong under OpenVINO. SmolVLA has a second graph builder used when a caller asks for per-phase timings, and it does not survive translation - max|delta| 1.9 on every device. The default TimingDetail::NONE path, which is what vla-server and vla-cli use, is correct, and on the native CPU backend the two paths agree exactly. Evo-1 and π0.5 are unaffected on the same path, so this is specific to SmolVLA's second graph; one hypothesis - that the split-graph guard sends it down the LLM path - was tested and is wrong. Per-stage timings for SmolVLA are omitted from the tables above.

TODO

Fixing issue channel not aligned by 16. Feasible solution is padding dummy channel so that number of channels is a multiple of 16.

Splitting across devices. Intel's own π0.5 write-up puts the vision encoder and language model on the iGPU and the action expert on the NPU, with the KV cache as the only cross-device handoff. The toolchain does not carry over - PyTorch exported to OpenVINO IR as three separate models, no ggml - but the shape of the answer does: the two devices suit different stages, and π0.5 on the NPU alone is already within 1.5x of the iGPU at a fraction of the power. vla.cpp cannot make that split today because the core drives one backend for a whole prediction. It would need a per-stage backend rather than a per-op scheduler - the vision tower, the prefix and the action expert already hand off through host memory, so the seam is in the right place - but that is an engine change, not a backend one.