QNN-delegated static-cache decoder produces correct output via native C++ Module, but degenerate/repeating output via Android JNI Module (same .pte, same device)

[aramvyan]

QNN-delegated static-cache decoder produces degenerate output through Android JNI Module, but correct output through the native C++ Module/runner — same .pte, same device

Summary

A Whisper .pte exported with to_edge_transform_and_lower_to_qnn() (QNN HTP backend, static KV-cache decoder) produces correct, real transcripts when run through ExecuTorch's native C++ Module API (examples/qualcomm/oss_scripts/whisper/runner/), but produces degenerate, repetition-loop garbage when run through the Android JNI Module/Tensor/EValue wrapper (extension/android/) — same .pte file (verified byte-identical via sha256), same physical device, same QNN SDK version, same greedy-decode algorithm (verified line-by-line equivalent).

Environment

• ExecuTorch: checked out fresh (commit at time of testing — pin to whatever was HEAD on the build box; not recorded precisely, should be captured before filing)
• QNN SDK: 2.37.0.250724
• Device: nubia NX789J-UN (Snapdragon 8 Elite Gen5, SM8750), Android 15
• HTP: v79 (libQnnHtpV79Skel.so/libQnnHtpV79Stub.so)
• Model: Whisper tiny, exported via examples/qualcomm/oss_scripts/whisper/whisper.py (16a8w quantization, PT2E, MinMaxObserver, per-channel conv+linear)

What works: native C++ path

Built qnn_whisper_runner via the official backends/qualcomm/scripts/build.sh --skip_x86_64, pushed to the device with the matching QNN runtime .sos (libQnnHtp.so, libQnnHtpV79Stub.so, libQnnHtpPrepare.so, libQnnSystem.so, libQnnHtpV79Skel.so), ran directly via adb shell (bare shell UID):

./qnn_whisper_runner --model_path whisper_tiny_qnn_16a8w.pte --tokenizer_json_path tokenizer.json \
  --input_list_path input_list.txt --seq_len 32 --output_folder_path outputs

input_0_0.raw was a [1,80,3000] float32 mel tensor computed via transformers.AutoProcessor on an 11s English speech clip (JFK inaugural excerpt).

Result (outputs/output_0.txt):

<|en|><|transcribe|><|notimestamps|> And so my fellow Americans ask not what your country can do
for you ask what you can do for your country.<|endoftext|>

27 tokens, real varying vocabulary throughout, correct structural token at position 3
(<|notimestamps|>, id 50363), clean EOS termination. Model load 0.44s, encode 0.032s, decode
0.159s — confirmed real HTP execution via [INFO] [Qnn ExecuTorch] logs, not a CPU fallback.

(Note: the two forced-token positions, <|en|>/<|transcribe|> at decoder positions 1/2, are an
application-level convention from the consuming Android app, replicated in the runner via a small
patch to runner.cpp for apples-to-apples comparison with the Android path below — not upstream
behavior. The important part is what happens from position 3 onward, which is entirely the model's
own unforced greedy output either way.)

What's broken: Android JNI path

Same .pte (verified byte-identical, sha256 match), loaded via org.pytorch.executorch.Module from
Kotlin:

val module = Module.load(modelFile.absolutePath, Module.LOAD_MODE_FILE)
module.loadMethod("encoder")
module.loadMethod("decoder")
// ... greedy decode loop, one execute() call per token:
val logits = module.execute(
    "decoder",
    EValue.from(Tensor.fromBlob(longArrayOf(curToken), longArrayOf(1, 1))),
    EValue.from(Tensor.fromBlob(attentionMask.copyOf(), longArrayOf(1, 1, 1, maxContextLen))),
    encoderOutValue,
    EValue.from(Tensor.fromBlob(longArrayOf(pos), longArrayOf(1))),
)[0].toTensor().dataAsFloatArray

Algorithm (attention-mask indexing, pos tracking, argmax selection) verified line-by-line
equivalent to the C++ runner's loop. Same device, same .pte, run moments apart.

Result, every run, all 6 Whisper sizes tested (tiny/base/small/medium/large-v3/large-v3-turbo):
degenerate output starting at decoder position 3 (the first position not forced by the calling
app), typically a repetition loop of a single token within 2-5 total generated tokens. Examples
observed on tiny across many runs on the same fixture: <|startoftranscript|> repeating,
<|sv|> repeating, <|transcribe|> repeating (varies run to run, and varied with different build
configurations — see below — but never produces real content). One data point of note:
large_v3's first free prediction was the correct structural token, <|notimestamps|> — then it
got stuck repeating that same token instead of advancing, rather than producing an implausible
token outright.

What we ruled out (9 independent hypotheses, each tested by actually rebuilding/reinstalling and

running on real hardware, not just reasoned about)

1. Calibration data quality/representativeness — an A/B between the model's original
   (unpatched, fully free-running) calibration and a corrected, properly-forced-token calibration
   produced byte-for-byte identical failure. Calibration data is not the cause.
2. Calibration missing a repetition-early-stop (unlike the consuming app's own decode loop,
   calibration only breaks on EOS) — patched calibration to add the same repetition-stop, re-ran
   full export + on-device test. No change.
3. skip_mutable_buffer (to_edge_transform_and_lower_to_qnn()'s QNN-partitioner flag,
   documented for exactly "accuracy issue during the runtime") — tested True vs default False.
   No change.
4. Module load mode — tested LOAD_MODE_MMAP_USE_MLOCK_IGNORE_ERRORS (matching the native
   runner's Module::LoadMode::MmapUseMlockIgnoreErrors exactly) vs the app's original
   LOAD_MODE_FILE. No change.
5. Single shared Module for encoder+decoder vs two independent Module instances (matching
   the native runner's WhisperEncoder/WhisperDecoder two-object split exactly, each loading the
   same .pte independently). No change.
6. On-device mel-spectrogram computation (the app's own whisper.cpp-based
   whisper_pcm_to_mel) vs the Python AutoProcessor-computed features used in the native-runner
   test — dumped the app's real computed mel tensor and compared numerically against the Python
   one: max abs diff 1.88e-5, correlation 0.99999999999981 (float32 noise only). Not the cause.
7. QNN runtime .so version mismatch — the app's bundled libQnnHtp.so/
   libQnnHtpPrepare.so/libQnnHtpV79Skel.so/libQnnHtpV79Stub.so are sha256-identical to the
   ones used in the successful native-runner test. Not the cause.
8. A stale/outdated executorch-android AAR — rebuilt the AAR from scratch via the official
   scripts/build_android_library.sh against the same checkout used for the successful native
   runner (installed a full Android SDK inside the same build environment for this). The freshly
   built libqnn_executorch_backend.so came out byte-identical to the currently-vendored one
   (build-configuration-deterministic, not stale). libexecutorch.so did differ from the
   currently-vendored one. Swapped the full freshly-built matched pair into the app, rebuilt,
   reinstalled, tested: behavior measurably changed (different specific repeating token, 4
   generated tokens instead of 5) but the core symptom — immediate degenerate output starting right
   after the two forced positions — persisted.
9. Zero-copy JNI tensor lifetime — extension/android/jni/jni_layer.cpp's
   TensorHybrid::newJTensorFromTensor() wraps the output tensor's data_ptr() directly in a
   zero-copy Java ByteBuffer (facebook::jni::JByteBuffer::wrapBytes), with an explicit code
   comment: safe only "as long as ... there is no other inference call" before the Java side reads
   it. The app's decode loop holds and reuses the encoder method's output Tensor across every
   subsequent decoder execute() call in the loop (i.e., many "other inference calls" after the
   one that produced it) — exactly the scenario the comment warns about. Deep-copied the encoder
   output into an independently-owned Kotlin FloatArray/Tensor before the decode loop instead of
   holding the zero-copy view. No change.

Also checked and ruled out: cross-thread execution (the consuming app runs model-load through every
encode/decode call on one dedicated single-thread Executor — never multi-threaded).

What we could not cleanly test

Tried running the native qnn_whisper_runner binary under the consuming app's own UID/SELinux
domain (adb shell run-as <pkg> ..., since /data/local/tmp binaries can't execute under
untrusted_app SELinux) to see whether HTP/FastRPC transport behaves differently there than under
bare shell. This hit a different failure (QnnDsp <E> Failed to load skel, error: 4000 /
Failed to create transport for device, error: 4000) regardless of which native-library directory
was used (including the app's real, properly-installed nativeLibraryDir) — but the real,
production Android app does get genuine HTP delegate access every single run (confirmed via
MethodMetadata.getBackends() reporting QnnBackend, and real NPU-speed timings, never a CPU
fallback signature), so this specific run-as-executed-binary test isn't a valid apples-to-apples
comparison — it's very likely missing some Android-lifecycle-level FastRPC/vendor-binder
registration step that only happens through normal Context/Application initialization, not
through a bare run-as exec. Abandoned this angle as inconclusive rather than chase a second,
apparently-unrelated failure mode.

Ask

Is there a known difference between how the native Module/backend delegate handles are
initialized/persisted versus how extension/android's JNI wrapper does, specifically for models
using QNN-delegated mutable buffers (static KV-cache decoders)? The QNN partitioner's own
skip_mutable_buffer=False (default) code comment documents a "same I/O address" optimization for
delegated mutable buffers during the runtime — is there anything about crossing the JNI boundary
(tensor construction via facebook::jni::JByteBuffer, Module object lifetime under ART's GC,
thread-affinity assumptions) that could break that address-stability assumption in a way a bare C++
call stack wouldn't?

Happy to provide the exact .pte, a minimal reproduction Kotlin/JNI harness, or run additional
diagnostics on request — this has been isolated fairly thoroughly on the calling-app side (9
independent hypotheses tested by actually rebuilding and running on real hardware, not just static
analysis) and increasingly looks like it needs an answer from inside ExecuTorch's own JNI/QNN
backend internals rather than anything fixable from the Android app or export-script side.

cc @cbilgin @psiddh

[shewu-quic]

Hi @aramvyan,
Thanks for the detailed analysis and for taking the time to investigate this issue. Based on your findings, it doesn't seem to be a problem in the QNN backend itself, since the example script produces the correct output.

As a next step, would it be possible to further narrow down where the accuracy issue is introduced with jni interface? For example, can we determine whether it already exists in the encoder output, or if it only appears in the decoder path?

I'm asking because the decoder model is the only part that uses skip_mutable_buffer, so understanding whether the issue is encoder-specific or decoder-specific could help further investigation.