Skip to content

Commit f7d2522

Browse files
authored
refactor and get same performance as Pytorch grap capture by using bf16 format (#21)
1 parent e9e57ea commit f7d2522

84 files changed

Lines changed: 6939 additions & 3549 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

CHANGELOG.md

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,50 @@
22

33
Notable changes to vla.cpp. Format loosely follows [Keep a Changelog](https://keepachangelog.com).
44

5+
## [0.3.0] - 2026-08-14
6+
7+
Every architecture is byte-identical to 0.2.0 at matching settings.
8+
`libero_object`, 100 episodes per model, on one RTX 3090:
9+
10+
| Model | SR | Latency, fastest | Fastest flags |
11+
|---|---:|---:|---|
12+
| `bitvla` | 99/100 | 48.0 ms | `--weight-dtype bf16` |
13+
| `gr00t_n1_5` | 99/100 | 67.9 ms | *(none)* |
14+
| `gr00t_n1_7` | 98/100 | 55.4 ms | *(none)* |
15+
| `openvla_oft` | 97/100 | 219.5 ms | *(none)* |
16+
| `pi05` | 96/100 | 112.3 ms | *(none)* |
17+
| `vla_adapter` | 96/100 | 69.7 ms | *(none)* |
18+
| `evo1` | 91/100 | 114.1 ms | `--act-dtype bf16 --flash-attn` |
19+
| `smolvla` | 90/100 | 50.5 ms | `--flash-attn --mm-prec default` |
20+
| `gr00t_n1_6` | 84/100 | 55.5 ms | *(none)* |
21+
| `pi0` | 81/100 | 94.1 ms | `--act-dtype bf16 --flash-attn` |
22+
| `vla_jepa` | not evaluated | 44.0 ms | *(none)* |
23+
24+
SR is measured at each model's defaults, so it does not carry over to the four
25+
rows whose fastest flags change numerics. Full detail in `refactor-report.md`.
26+
27+
### Added
28+
- Model code split into three levels: `src/layers/` (stateless graph fragments), `src/modules/` (weights plus the graph consuming them), `src/models/` (config, composition, `predict`).
29+
- `vla::WeightLoader`: declares weights by name, reports a miss once, allocates and uploads in one call. Replaces the `mk`/`mk_mm`/`mk_f32` lambdas and `ok &= a&&b&&c` chain each of the eleven architectures carried.
30+
- `vla-server` flags `--weight-dtype f32|bf16`, `--act-dtype f32|bf16`, `--flash-attn [0|1]`, `--mm-prec default|f32`, also readable from a `"runtime"` object in the `--config` JSON.
31+
- `eval/refactor_verify.sh`: diffs every architecture's action chunk at two precisions against a reference run, with `BENCH=N` for per-config `predict()` timing.
32+
33+
### Changed
34+
- GR00T N1.5/N1.6/N1.7 and VLA-JEPA default to BF16 weights. `--weight-dtype f32` restores the old default of v0.2.0, bit-identically.
35+
- The per-architecture precision switches (`VLA_GR00T_BF16_WEIGHTS`, `VLA_*_FA`, `VLA_*_BF16_ACT`, `VLA_*_F32_WEIGHTS`, `VLA_MM_PREC`, `VLA_WEIGHT_DTYPE`) are retired. Setting one now fails the load naming its replacement instead of being ignored.
36+
- Deduplicated: `build_dit_block` 4 copies to 1, `SigLipLayerW` 5 to 1, `Qwen3LayerW` 4 to 1, the DINOv2+SigLIP declaration 2 to 1. 1,817 lines of shared code now serve all eleven architectures.
37+
- BF16 elementwise kernels address rows by block index instead of a per-element 64-bit divide, and move eight values per thread on contiguous rows. `VLA_BF16_FLAT=1` selects the scalar path.
38+
39+
### Fixed
40+
- BF16 activations aborted on any fused elementwise run: ggml fuses upstream of the extension hook, and its fused path handles F32/F16 only. The hook now gets first refusal on fused add/mul.
41+
- Boolean environment switches read their value, not their presence, so `VLA_EVO1_FA=0` no longer enabled flash attention.
42+
- SmolVLA ignored its runtime options, leaving its weight dtype unsettable once `VLA_WEIGHT_DTYPE` retired.
43+
44+
45+
### Known limitations
46+
- Thread count, solver steps, GR00T embodiment and un-normalisation key remain environment-only (`VLA_N_THREADS`, `VLA_NUM_STEPS`, `VLA_GR00T_EMBODIMENT`, `VLA_*_UNNORM_KEY`).
47+
- VLA-JEPA has no LIBERO success rate; the client cannot emit its `<embodied>` tokens.
48+
549
## [0.2.0] - 2026-08-12
650

751
### Added

CMakeLists.txt

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,10 +36,15 @@ if(GGML_CUDA)
3636
set(_vla_llama_patch PATCH_COMMAND ${Python3_EXECUTABLE}
3737
${CMAKE_CURRENT_SOURCE_DIR}/scripts/patch_ggml_cuda_ext_hook.py <SOURCE_DIR>)
3838
endif()
39+
# Overridable so a regression can be bisected against another tag in a separate
40+
# build dir (-DVLA_LLAMA_TAG=b10326) without editing this file. The patch
41+
# anchors in scripts/patch_ggml_cuda_ext_hook.py are checked against the default.
42+
set(VLA_LLAMA_TAG "b10331" CACHE STRING "llama.cpp tag to fetch")
43+
3944
include(FetchContent)
4045
FetchContent_Declare(llama
4146
GIT_REPOSITORY https://github.com/ggml-org/llama.cpp
42-
GIT_TAG b10331
47+
GIT_TAG ${VLA_LLAMA_TAG}
4348
GIT_SHALLOW TRUE
4449
${_vla_llama_patch}
4550
)
@@ -64,6 +69,14 @@ vla_exclude_fetched_targets(${llama_SOURCE_DIR})
6469

6570
add_library(vla_core
6671
src/model.cpp
72+
src/loader.cpp
73+
src/options.cpp
74+
src/modules/action_expert.cpp
75+
src/modules/dit_head.cpp
76+
src/modules/encoder.cpp
77+
src/modules/prompt.cpp
78+
src/modules/qwen3_lm.cpp
79+
src/modules/siglip_vit.cpp
6780
src/models/smolvla.cpp
6881
src/models/pi0.cpp
6982
src/models/pi05.cpp
@@ -114,6 +127,9 @@ if(GGML_CUDA)
114127
CUDA_SEPARABLE_COMPILATION ON
115128
POSITION_INDEPENDENT_CODE ON
116129
)
130+
target_include_directories(bitvla_cuda_kernels PRIVATE
131+
${CMAKE_CURRENT_SOURCE_DIR}/src
132+
)
117133
target_compile_features(bitvla_cuda_kernels PRIVATE cxx_std_17)
118134
target_compile_options(bitvla_cuda_kernels PRIVATE
119135
$<$<COMPILE_LANGUAGE:CUDA>:-O3 --use_fast_math -Xptxas=-O3>

README.md

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,16 @@ Use `--bind` to change the address and port. Stop the server with `Ctrl-C`.
153153

154154
`vla-server` also takes `-hf user/repo[:file.gguf]` in place of a checkpoint path.
155155

156+
Precision flags (`vla-server --help` for the full list).
157+
The fastest configuration per model, with measured latency and success rate, is
158+
in [`CHANGELOG.md`](CHANGELOG.md):
159+
160+
- `--weight-dtype f32|bf16` - resident dtype for GEMM weights.
161+
- `--act-dtype f32|bf16` - activation dtype; needs CUDA and bf16 weights.
162+
- `--flash-attn` - faster on the larger towers, but changes numerics.
163+
- `--mm-prec default|f32` - matmul accumulation precision.
164+
165+
156166
Environment knobs that apply to every arch:
157167

158168
- `VLA_N_THREADS` - CPU backend thread count, default core count capped at 16.
@@ -180,14 +190,14 @@ python eval/client/run_sim_client_direct.py \
180190
The GR00T models need two extras:
181191

182192
- client side: `--stats-json /path/to/dataset_statistics.json`
183-
- server side: `VLA_GR00T_EMBODIMENT` (`new_embodiment` for N1.5, `libero_panda` for N1.6, `libero_sim` for N1.7) and `VLA_GR00T_BF16_WEIGHTS=1` (to fit the 8 GB card).
193+
- server side: `VLA_GR00T_EMBODIMENT` (`new_embodiment` for N1.5, `libero_panda` for N1.6, `libero_sim` for N1.7).
184194

185195
### SimplerEnv
186196

187197
So far only **GR00T-N1.6** is wired (the `gr00t-n1d6-bridge` checkpoint with the `oxe_widowx` embodiment). Start `vla-server` on port 5566 with `oxe_widowx` embodiment:
188198

189199
```bash
190-
VLA_GR00T_BF16_WEIGHTS=1 VLA_GR00T_EMBODIMENT=oxe_widowx \
200+
VLA_GR00T_EMBODIMENT=oxe_widowx \
191201
./build/vla-server "$GR00T_N1D6_GGUF"
192202
```
193203

ci/lib/common.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,7 +139,7 @@ apply_gr00t_env() {
139139
local arch="$1"
140140
case "$arch" in
141141
gr00t_n1_5|gr00t_n1_6|gr00t_n1_7)
142-
export VLA_GR00T_BF16_WEIGHTS="${VLA_GR00T_BF16_WEIGHTS:-1}" ;;
142+
: ;;
143143
esac
144144
case "$arch" in
145145
gr00t_n1_5) export VLA_GR00T_EMBODIMENT="${VLA_GR00T_EMBODIMENT:-new_embodiment}" ;;

docs/backend/sycl.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -217,6 +217,6 @@ weights) does not fit and dies in the allocator:
217217
level_zero backend failed with error: 38 (UR_RESULT_ERROR_OUT_OF_HOST_MEMORY)
218218
```
219219

220-
`VLA_GR00T_BF16_WEIGHTS=1` halves the weights but its activations still overflow
220+
`--weight-dtype bf16` (now the default) halves the weights but its activations still overflow
221221
the card. There is no host-memory spill path - the core is single-backend - so
222222
the larger checkpoints need an A770/B580-class card or better.

eval/README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ Notes:
5656
accept the licence, or point `--tokenizer` at a local copy.
5757
- **GR00T** arches need `--stats-json <ckpt>/dataset_statistics.json` and an embodiment selected
5858
server-side via `VLA_GR00T_EMBODIMENT` (`new_embodiment` for N1.5, `libero_panda` for N1.6,
59-
`libero_sim` for N1.7), plus `VLA_GR00T_BF16_WEIGHTS=1` to fit an 8 GB card.
59+
`libero_sim` for N1.7). BF16 weights are the default, which is also what fits an 8 GB card.
6060

6161
To sweep every model over `libero_object` tasks 0–9, use `eval/run_libero.sh -i <MODELS_ROOT>`.
6262

@@ -66,7 +66,7 @@ So far only **GR00T-N1.6** is wired (the `gr00t-n1d6-bridge` checkpoint with the
6666
embodiment). Serve it, then drive from the SimplerEnv venv:
6767

6868
```bash
69-
VLA_GR00T_BF16_WEIGHTS=1 VLA_GR00T_EMBODIMENT=oxe_widowx \
69+
VLA_GR00T_EMBODIMENT=oxe_widowx \
7070
./build/vla-server "$GR00T_N1D6_GGUF"
7171

7272
eval/sim/simpler/simpler_uv/.venv/bin/python eval/client/run_simpler_client_direct.py \

eval/refactor_verify.sh

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
#!/usr/bin/env bash
2+
# Copyright 2026 VinRobotics - Apache-2.0
3+
#
4+
# Bit-exactness and latency harness for the src/ layer/module/model refactor.
5+
#
6+
# eval/refactor_verify.sh <outdir> actions only
7+
# BENCH=20 eval/refactor_verify.sh <outdir> actions + predict() timing
8+
#
9+
# Each arch runs twice: at its shipping defaults, and under the alternate
10+
# precision. Both must stay byte-identical across a refactor, and neither may
11+
# regress in latency.
12+
#
13+
# eval/refactor_verify.sh outputs/refactor/before
14+
# ...change...
15+
# cmake --build build -j"$(nproc)" --target vla_predict_check
16+
# eval/refactor_verify.sh outputs/refactor/after
17+
# diff -r outputs/refactor/before outputs/refactor/after
18+
#
19+
# Never rebuild while a sweep is running: relinking libvla_core.so under it
20+
# makes every remaining arch fail to load.
21+
#
22+
# ARCHS=... restricts the sweep. The square input side is probed rather than
23+
# hardcoded, because a tower fed the wrong side returns action_len=0 instead of
24+
# failing, and a wrong side would silently "pass" a diff.
25+
26+
set -euo pipefail
27+
28+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
29+
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
30+
31+
BIN="${BIN:-${REPO_ROOT}/build/tests/vla_predict_check}"
32+
HF="${HF:-/mnt/data/hf_data/vrfai}"
33+
OUT="${1:-${REPO_ROOT}/outputs/refactor/baseline}"
34+
SIDES="${SIDES:-224 256 448 512}"
35+
BENCH="${BENCH:-0}"
36+
export CUDA_VISIBLE_DEVICES="${CUDA_VISIBLE_DEVICES:-0}"
37+
38+
# arch|ckpt|mmproj|n_images|env|alternate-config CLI flags.
39+
# openvla_oft has no alternate: at f32 its weights need 30 GB.
40+
MODELS=(
41+
"smolvla|${HF}/smolvla-libero-gguf/smolvla-libero.gguf|${HF}/backup/mmproj-smolvla-libero.gguf|2||--flash-attn --mm-prec default"
42+
"pi0|${HF}/pi0-libero-finetuned-v044-gguf/pi0-libero-finetuned-v044.gguf|${HF}/backup/mmproj-pi0-libero-finetuned-v044.gguf|2||--act-dtype bf16 --flash-attn"
43+
"pi05|${HF}/pi05-libero-gguf/pi05-libero.gguf|${HF}/backup/mmproj-pi05-libero.gguf|2||--weight-dtype f32"
44+
"evo1|${HF}/evo1-libero-gguf/evo1-libero.gguf||2||--act-dtype bf16 --flash-attn"
45+
"gr00t_n1_5|${HF}/gr00tn1d5-libero-object-gguf/gr00tn1d5-libero-object.gguf||2||--weight-dtype f32"
46+
"gr00t_n1_6|${HF}/gr00tn1d6-libero-gguf/gr00tn1d6-libero.gguf||2||--weight-dtype f32"
47+
"gr00t_n1_7|${HF}/gr00tn1d7-libero-gguf/libero_object/gr00tn1d7-libero-object.gguf||2||--weight-dtype f32"
48+
"bitvla|${HF}/bitvla-libero-gguf/libero_object/bitvla-libero-object.gguf||2||--weight-dtype bf16"
49+
"vla_adapter|${HF}/vla-adapter-libero-object-gguf/libero_object/vla-adapter-libero-object.gguf||2||--weight-dtype f32"
50+
"openvla_oft|${HF}/openvla-oft-libero-gguf/openvla-oft-libero.gguf||2||"
51+
"vla_jepa|${HF}/vla-jepa-libero/vla-jepa.gguf||2|VLA_EXTRA_TOKEN=151697 VLA_EXTRA_COUNT=32|--weight-dtype f32"
52+
)
53+
54+
[[ -x "${BIN}" ]] || { echo "ERROR: missing ${BIN} (cmake -DVLA_BUILD_TESTS=ON)" >&2; exit 1; }
55+
mkdir -p "${OUT}"
56+
57+
run_one() {
58+
local arch="$1" ckpt="$2" mmproj="$3" nimg="$4" env_str="$5" tag="$6" side="$7" cli="$8"
59+
# shellcheck disable=SC2086
60+
env ${env_str} VLA_IMG_SIZE="${side}" VLA_BENCH_ITERS="${BENCH}" \
61+
"${BIN}" "${ckpt}" "${mmproj}" "${nimg}" ${cli} \
62+
> "${OUT}/${arch}${tag}.actions.txt" 2> "${OUT}/${arch}${tag}.log"
63+
}
64+
65+
fail=0
66+
for row in "${MODELS[@]}"; do
67+
IFS='|' read -r arch ckpt mmproj nimg always fastest <<< "${row}"
68+
69+
if [[ -n "${ARCHS:-}" && " ${ARCHS} " != *" ${arch} "* ]]; then
70+
continue
71+
fi
72+
if [[ ! -e "${ckpt}" ]]; then
73+
echo "[skip] ${arch}: no checkpoint at ${ckpt}"
74+
continue
75+
fi
76+
77+
side=""
78+
for s in ${SIDES}; do
79+
if run_one "${arch}" "${ckpt}" "${mmproj}" "${nimg}" "${always}" "" "${s}" "" \
80+
&& ! grep -q '^action_len=0$' "${OUT}/${arch}.actions.txt"; then
81+
side="${s}"
82+
echo "${s}" > "${OUT}/${arch}.side"
83+
break
84+
fi
85+
done
86+
if [[ -z "${side}" ]]; then
87+
echo "[FAIL] ${arch}: no input side in '${SIDES}' produced a chunk; see ${OUT}/${arch}.log" >&2
88+
fail=1
89+
continue
90+
fi
91+
92+
line="[ok ] ${arch} side=${side}"
93+
[[ "${BENCH}" -gt 0 ]] && line+=" default=$(grep -oP 'min=\K[0-9.]+' "${OUT}/${arch}.log" | head -1)ms"
94+
95+
if [[ -n "${fastest}" ]]; then
96+
if run_one "${arch}" "${ckpt}" "${mmproj}" "${nimg}" "${always}" ".alt" "${side}" "${fastest}" \
97+
&& ! grep -q '^action_len=0$' "${OUT}/${arch}.alt.actions.txt"; then
98+
line+=" alt=ok"
99+
[[ "${BENCH}" -gt 0 ]] && line+=" $(grep -oP 'min=\K[0-9.]+' "${OUT}/${arch}.alt.log" | head -1)ms"
100+
else
101+
echo "[FAIL] ${arch}: alternate config produced no chunk; see ${OUT}/${arch}.fast.log" >&2
102+
fail=1
103+
fi
104+
fi
105+
echo "${line}"
106+
done
107+
108+
echo
109+
echo "written to ${OUT}"
110+
exit "${fail}"

eval/run_libero.sh

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -298,9 +298,10 @@ run_model() {
298298
client_extra+=(--stats-json "${stats_json}")
299299
fi
300300

301+
# BF16 weights are the shipping default now, and are what every published
302+
# GR00T success rate was measured under; passed explicitly so the log records it.
301303
if [[ "${arch}" == gr00t_n1_5 || "${arch}" == gr00t_n1_6 || "${arch}" == gr00t_n1_7 ]]; then
302-
export VLA_GR00T_BF16_WEIGHTS="${VLA_GR00T_BF16_WEIGHTS:-1}"
303-
echo "[${arch}] VLA_GR00T_BF16_WEIGHTS=${VLA_GR00T_BF16_WEIGHTS}"
304+
server_args+=(--weight-dtype "${WEIGHT_DTYPE:-bf16}")
304305
fi
305306
if [[ -n "${_USER_VLA_GR00T_EMBODIMENT}" ]]; then
306307
export VLA_GR00T_EMBODIMENT="${_USER_VLA_GR00T_EMBODIMENT}"

eval/run_libero_client.sh

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -142,21 +142,20 @@ EOF
142142
;;
143143
gr00t_n1_5)
144144
cat <<EOF
145-
VLA_GR00T_BF16_WEIGHTS=1 VLA_GR00T_EMBODIMENT=new_embodiment \\
145+
VLA_GR00T_EMBODIMENT=new_embodiment \\
146146
./build/vla-server --bind tcp://*:${PORT} \\
147147
\${MODELS_ROOT}/gr00tn1d5-libero-object-gguf/gr00tn1d5-libero-object.gguf
148148
EOF
149149
;;
150150
gr00t_n1_6)
151151
cat <<EOF
152-
VLA_GR00T_BF16_WEIGHTS=1 VLA_GR00T_EMBODIMENT=libero_panda \\
152+
VLA_GR00T_EMBODIMENT=libero_panda \\
153153
./build/vla-server --bind tcp://*:${PORT} \\
154154
\${MODELS_ROOT}/gr00t-n1d6-libero-gguf/gr00t-n1d6-libero.gguf
155155
EOF
156156
;;
157157
gr00t_n1_7)
158158
cat <<EOF
159-
VLA_GR00T_BF16_WEIGHTS=1 \\
160159
./build/vla-server --bind tcp://*:${PORT} \\
161160
\${MODELS_ROOT}/gr00t-n1d7-libero-object-gguf/gr00t-n1d7-libero-object.gguf
162161
EOF

eval/run_libero_server.sh

Lines changed: 3 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ Usage: $(basename "$0") -m MODEL [-i MODELS_ROOT] [-a BIND_ADDR] [-o LOG_DIR] [-
4545
-B skip the cmake build (use the existing vla-server binary)
4646
-h show this help
4747
48-
Env overrides: BIND_ADDR, SERVER_BIN, VLA_GR00T_BF16_WEIGHTS, VLA_GR00T_EMBODIMENT
48+
Env overrides: BIND_ADDR, SERVER_BIN, VLA_GR00T_EMBODIMENT
4949
EOF
5050
}
5151

@@ -143,13 +143,9 @@ case "${MODEL}" in
143143
esac
144144

145145
# GR00T env. Honour a user-supplied VLA_GR00T_EMBODIMENT; otherwise default per
146-
# arch (matches eval/run_libero.sh). VLA_GR00T_BF16_WEIGHTS defaults to 1 (BF16 is
147-
# the smaller weight path - important on the Nano's 8 GB unified RAM).
146+
# arch (matches eval/run_libero.sh). BF16 weights are the shipping default and
147+
# the smaller weight path - important on the Nano's 8 GB unified RAM.
148148
_USER_VLA_GR00T_EMBODIMENT="${VLA_GR00T_EMBODIMENT-}"
149-
if [[ "${MODEL}" == gr00t_n1_5 || "${MODEL}" == gr00t_n1_6 || "${MODEL}" == gr00t_n1_7 ]]; then
150-
export VLA_GR00T_BF16_WEIGHTS="${VLA_GR00T_BF16_WEIGHTS:-1}"
151-
echo "[${MODEL}] VLA_GR00T_BF16_WEIGHTS=${VLA_GR00T_BF16_WEIGHTS}"
152-
fi
153149
if [[ -n "${_USER_VLA_GR00T_EMBODIMENT}" ]]; then
154150
export VLA_GR00T_EMBODIMENT="${_USER_VLA_GR00T_EMBODIMENT}"
155151
else

0 commit comments

Comments
 (0)