GPU backend bug?

[BmanClark]

I have a model that runs on CPU, but gets incorrect results on GPU. The model is float32 (quantization would be the next step), and while I've adjusted it to remove non-GPU supported operators, and reduce 5D tensors to 4D, the CPU results are still identical, so I'm confident that the model is fine.

I am using LiteRT 2.1.0 with the Kotlin API, which says I shouldn't have to do anything different with the buffers than CPU, even though it creates OpenClBufferPacked Buffers rather than HostMemory ones.
I was running through the same code for CPU/GPU (just with different accelerator option), but I've adjusted to try and eliminate any source of issues - forcing precision to FP32, doing everything from model creation to results in a single thread in a single function, and adjusting how I write to the buffers. The model should have dynamic input, but I'm now padding with 0s to be safe. I still get consistently the same outputs for given inputs for both CPU and GPU through all the tests. Indeed if I make a one-character change to the below code to choose CPU accelerator I get the correct result. GPU consistently gives near-zero (but not zero) outputs

private val gpuThread = Executors.newSingleThreadExecutor()

fun runOnGpu(audio: FloatArray, cb: (FloatArray) -> Unit) {
  gpuThread.execute {
    val opts = CompiledModel.Options(toAccelerator(AcceleratorEnum.GPU)).apply {
        gpuOptions = CompiledModel.GpuOptions(
            precision = CompiledModel.GpuOptions.Precision.FP32
        )
    }
    val model = CompiledModel.create(
        context.assets,
        "$name.tflite",
        opts,
        null
    )
    val inputs = model.createInputBuffers()
    val outputs = model.createOutputBuffers()

    inputs[0].writeFloat(prep(audio))   // must be exact length
    model.run(inputs, outputs)
    val logits = outputs[0].readFloat()

    // close everything if you recreate per run
    outputs.forEach { it.close() }
    inputs.forEach { it.close() }
    model.close()

    cb(logits)
  }
}

So I suspect a GPU backend bug, unless there is something undocumented (or at least I've not found) about buffer input/output.

The .tflite file in question is 467MB, but I can upload it somewhere.
The model contains operators: Conv2D, FullyConnected, Transpose, Batch Matrix Multiply, GELU, ADD, DEPTHWISE_CONV_2D, Multiply, Softmax, Subtract, Mean, Slice, Squared Difference, Logistic, Pad, Reciprocal Square Root, RESHAPE, Sum - do any of them have known issues?

[aravindmurali22]

Hi @BmanClark,

"Near-zero" GPU results often indicate precision issues, driver bugs, or layout mismatches (e.g. reading padding).

To help us investigate, could you please provide:

1. Exact model (e.g. Pixel 9 / Snapdragon 8 Gen 3).
2. A minimal .tflite that reproduces this behavior.
3. Does this same model work with the standard InterpreterApi GPU delegate on the same device?

[BmanClark]

Sorry, was off for a few days.
I'm using a Vivo X100 (Dimensity 9300).
I've managed to get the TFLite InterpreterApi GPU Delegate going (2.17.0), and it does get correct results (it's quite a bit slower, slower than CPU in fact, so at least new GPU backend has speed going for it). I get same results from TFLite interpreter CPU/GPU and LiteRT CPU, just LiteRT GPU is near zero.
Minimal tflite I'm less confident on working out. I can provide the whole tflite (plus example input/output) if you give me somewhere to put it, but I don't know about cutting it down to the failing part. I don't actually even have the original model, I'm loading from HuggingFace transformers library, and then monkey patching (significantly) to get it to be compatible with GPU delegate. (patches to: pre-compute a section that had several incompatible operators; re-do GroupNorm so convertor didn't create unneeded GATHER_NDs; and to reduce 5D Tensors in attention to 4D).
Model was originally:

model_path = "roblox/voice-safety-classifier-v2"

model = WavLMForSequenceClassification.from_pretrained(
    model_path, num_labels=len(labels_name_list)
)