Vulkan compute output corrupted on Imagination/PowerVR with opt.lightmode=1 blob recycling — allocator bookkeeping proven correct by internal tracing during a corrupted inference; barriers and submit boundaries ruled out

[di2g]

error log | 日志或报错信息 | ログ

No error is reported anywhere — that is part of the problem. load_param,
load_model and every extract() return 0, no validation/driver message, but the
extracted output of a RIFE v4.7 frame-interpolation net is corrupted ("melted"):
frames mathematically unrelated to the two input frames (our automated oracle: a
blended frame farther from BOTH endpoints than they are from each other — impossible
for any convex weighting of the inputs; ratios 2.9x–5.4x the endpoint separation).

The same net, same weights, same inputs:

• ncnn CPU path: bit-clean, always.
• ncnn Vulkan on ARM Mali / Qualcomm Adreno (6 devices in the field): clean.
• ncnn Vulkan on Imagination PowerVR (3 GPU generations, incl. DXT-48 /
  Pixel 10 Pro, vendorID 0x1010): corrupted, reproducibly.
• Same device, opt.lightmode=0: always clean (2/2) — at ~1.8x time and ~18x
  peak memory, as expected with no blob recycling.

So: PowerVR + lightmode=1 blob recycling is the triggering combination.

context | 编译/运行环境 | バックグラウンド

• ncnn tag 20260526, static lib, NCNN_VULKAN=ON, built with the release
  workflow's own CMake options (also reproduced with the official
  ncnn-20260526-android-vulkan.zip prebuilt — identical corruption), NDK
  27.2.12479018, arm64-v8a, Android app.
• Device: Google Pixel 10 Pro (Tensor G5), GPU Imagination PowerVR DXT-48,
  VkPhysicalDeviceProperties::vendorID = 0x1010, Android 17
  (google/blazer/blazer:17/CP2A.260805.005/15828068, ro.hardware.vulkan=powervr).
  Also observed on two older PowerVR generations on real-user devices.
• Net: RIFE v4.7 (583 layers / 752 blobs), buffer storage mode
  (use_image_storage off), fp32 (use_fp16_* off), elempack=1 on every dispatch
  (measured), 1920x1088 inputs, one custom layer (rife.Warp, warp/grid-sample —
  exonerated below). rough_score() on this device = 23 → pending-dispatch
  threshold 1M, which auto-flushes ~9-10 submit_and_wait() per extraction
  (measured; relevant to what is ruled out below).

how to reproduce | 复现步骤 | 再現方法

1. Any PowerVR-Vulkan Android device (vendorID 0x1010). Build a deep net whose blob
   lifetimes force VkBlobAllocator sub-buffer recycling (hundreds of transient
   blobs), opt.lightmode=1, buffer storage, fp32.
2. Run the same input through the CPU path and the Vulkan path; diff.
3. Vulkan output is corrupted; CPU is the reference. opt.lightmode=0 on the same
   device is clean, which isolates the trigger to blob recycling.

We can share our full deterministic repro (app + clip + forced inputs) privately if
useful; we have not yet reduced it to a standalone ~60-line .param chain — happy
to do so if a maintainer wants to pursue this. What a maintainer without a PowerVR
device would most need from us is runs of instrumented builds — we can run any
patch on the affected device and return traces.

more | 其他 | その他

We spent several sessions trying to localize this INSIDE ncnn before concluding the
defect is most likely in the PowerVR driver's handling of a legal-per-spec pattern
ncnn's allocator produces. Evidence, each item established by an on-device
experiment (not argument):

1. Allocator bookkeeping is CORRECT — proven by internal tracing DURING two
corrupted inferences (the strongest artifact here; traces + patch in the evidence gist below).
We built ncnn from source at the pin with instrumentation inside
VkBlobAllocator::fastMalloc/fastFree (patch in the evidence gist): a shadow interval set of
live hand-outs that flags any hand-out overlapping a live range, plus free-list
invariant validation (self-overlap, free-vs-live overlap) after every mutation, plus
per-release VkBufferMemory::refcount logging at net.cpp's lightmode release
sites. Result over two renders (~15,700 hand-outs, ~440 recycled ranges and ~570
refcount>1 releases per extraction): zero anomalies — no overlap-with-live, no
free-list corruption, no double-free — in BOTH runs, and both runs' outputs were
corrupted (allocdiag_cut173_melted.log and allocdiag_cut64_melted.log in the gist). Every recycled range was handed out only after its
final refcount-0 free. So the earlier suspicion that nested sub-buffer reuse
(allocator.cpp budget merging) was aliasing live data is REFUTED.

2. It is not the "erased hazard state" / missing-barrier issue either.
allocator.cpp:723-724 stamps recycled hand-outs access_flags=0 /
stage_flags=TOP_OF_PIPE (empty first sync scope for the first
barrier_readwrite). We tested a VkBlobAllocator subclass stamping conservative
flags (SHADER_READ|SHADER_WRITE / COMPUTE_SHADER) on every allocation, proven
active by logging: output bit-identical to baseline (identical corruption
ratios to 3 significant figures). command.cpp:2193-2194 already writes those
values after first use, so steady-state barriers are real — and making the
first-use barrier real too changes nothing.

3. It is not a synchronization-timing race ncnn could fix with more submits.
Explicit submit_and_wait(); reset(); inserted at 16 points per extraction (after
each Warp) still produces corrupted output (bounded "ghosting" variant). Note the
device also auto-flushes ~9-10 submit_and_wait() per extraction anyway
(rough_score cadence, net.cpp:250-272) — corruption survives real fence-waited
submission boundaries.

4. Layer-level suspects exonerated. The custom Warp layer's GPU-vs-CPU
self-test matches to 2.4e-06; forcing ONLY Warp to CPU (ncnn's own
download/upload path) is clean — but that inserts full submit+wait+copy, i.e. it
removes the recycling exposure, not a Warp bug. use_packing_layout=false
corrupts bit-identically (pack conversions exonerated). fp16 fully off.

5. Corruption has race-like signatures. A "delay reuse by K allocations"
allocator experiment changed WHICH content was corrupted and made output
nondeterministic across runs (from ghosting on one input to full-frame noise on
another) — the signature of re-rolling a scheduling race, not of a functional bug.
The corruption itself is deterministic per (build, input) — reproducible on demand.

Summary: with lightmode recycling ON, PowerVR intermittently mis-executes a
dispatch stream in which sub-ranges of one VkBuffer are freed and re-bound as
storage buffers across dispatches — while ncnn's bookkeeping, barriers, and even
hard submission boundaries are demonstrably in order. lightmode=0 (no recycling)
is the only clean configuration at GPU speed. We currently ship with ncnn-Vulkan
denied on vendorID 0x1010 as a mitigation.

Possible prior art: #5990 ("NCNN has different results based on CPU vs vulkan") —
intermittent corrupt Vulkan output, fp16 ruled out by the reporter, closed
unresolved; consistent with this class.

We are filing a parallel report with Imagination. From ncnn's side, the actionable
options we see: (a) a documented vendor workaround (e.g. treat vendorID 0x1010 as
lightmode-unsafe or add an option to disable sub-buffer recycling without
retaining every blob), or (b) help reducing this to a standalone repro the driver
team can run. We're glad to run instrumented builds on the affected device.

---

Evidence (public gist): https://gist.github.com/di2g/a3788246430458f8bad13c9308f18adf — instrumentation patch, both full allocator traces (each from a corrupted inference, zero anomalies), trace line-format key, and app-log excerpts proving the GPU path executed.

[di2g]

Follow-up: a middleware-level workaround is also refuted — further evidence the defect is driver-side.

We tested the obvious ncnn-side workaround for the pattern characterized above: an app-side VkAllocator subclass giving every blob its own VkBuffer at offset 0 with its own VkDeviceMemory (whole buffers recycled by exact size class — no storage-buffer descriptor ever aliases a sub-range of a shared buffer), set as both blob_vkallocator and workspace_vkallocator. RAM and speed stayed at lightmode=1 levels (PSS 1.13x, wall-clock 1.04x vs stock).

Result, in a same-session same-build A/B flipped by a runtime property:

• dedicated whole-buffer allocator → still corrupt, with a NEW signature (full-frame RGB channel-banded striping across the interpolated window; frames outside the window pristine);
• stock VkBlobAllocator control → classic grey-black melt.

Three allocation strategies (stock sub-range recycling, delayed/changed reuse geometry, dedicated whole-buffer-per-blob) now yield three distinct corruption signatures — same net, weights, inputs. So the PowerVR defect is not specific to sub-range aliasing: any recycling of GPU memory objects in this command stream corrupts, pattern-dependently. A zero-reuse configuration is untestable by construction (lightmode frees during command recording, so destruction must be deferred — and a deferral pool is reuse); lightmode=0 remains the one clean configuration at ~18x peak memory.

Implication for ncnn: an allocator-structure workaround is not available for this vendor — a vendor gate on vendorID 0x1010 (deny Vulkan, or force lightmode=0 where memory allows) looks like the only middleware-side mitigation.

New evidence in the same gist (https://gist.github.com/di2g/a3788246430458f8bad13c9308f18adf): dedicated_allocator.h (the full allocator, reusable as-is) and dedicated_allocator_ab_result.md (A/B details).