[Mobile] Onnxruntime react-native issue: [java.lang.ClassCastException: java.lang.String[][] cannot be cast to java.lang.String[]]

[MAProcessing]

Describe the issue

I'm developing a speech recognition mobile app, i'm using whisper onnx cpu-int8 generated with olive. I'm encountering a critical issue java.lang.ClassCastException: java.lang.String[][] cannot be cast to java.lang.String[] when running inference with onnxruntime-react-native in an Expo/React Native app. This occurs in session.run(feeds). I'm not sure if the problem source is the processed audio tensor or in onnxruntime or the output.
Here is the full log:
.../[lessonId].tsx:245 Inference problem : java.lang.ClassCastException: java.lang.String[][] cannot be cast to java.lang.String[]
at promiseMethodWrapper (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:2855:45)
at ?anon_0_ (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…ransform.routerRoot=app&unstable_transformProfile=hermes-stable:208499:120)
at next (native)
at asyncGeneratorStep (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:5548:19)
at next (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:5562:29)
at anonymous (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:5567:14)
at tryCallTwo (address at InternalBytecode.js:1:1222)
at doResolve (address at InternalBytecode.js:1:2541)
at Promise (address at InternalBytecode.js:1:1318)
at anonymous (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:5559:25)
at apply (native)
at run (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…transform.routerRoot=app&unstable_transformProfile=hermes-stable:208504:28)
at ?anon_0 (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…transform.routerRoot=app&unstable_transformProfile=hermes-stable:206799:47)
at next (native)
at asyncGeneratorStep (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:5548:19)
at next (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:5562:29)
at anonymous (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:5567:14)
at tryCallTwo (address at InternalBytecode.js:1:1222)
at doResolve (address at InternalBytecode.js:1:2541)
at Promise (address at InternalBytecode.js:1:1318)
at anonymous (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:5559:25)
at apply (native)
at run (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…transform.routerRoot=app&unstable_transformProfile=hermes-stable:206815:28)
at ?anon_0 (http://192.168.11.210:8081/app\lessons\learnings\[lessonId].bundle//&platfo…dev=true&minify=false&modulesOnly=true&runModule=false&shallow=true:265:44)
at next (native)
at asyncGeneratorStep (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:5548:19)
at _next (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:5562:29)
at tryCallOne (address at InternalBytecode.js:1:1180)
at anonymous (address at InternalBytecode.js:1:1874)
at apply (native)
at anonymous (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…&transform.routerRoot=app&unstable_transformProfile=hermes-stable:12801:50)
at _callTimer (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…&transform.routerRoot=app&unstable_transformProfile=hermes-stable:12725:17)
at _callReactNativeMicrotasksPass (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…&transform.routerRoot=app&unstable_transformProfile=hermes-stable:12755:17)
at callReactNativeMicrotasks (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…&transform.routerRoot=app&unstable_transformProfile=hermes-stable:12910:44)
at __callReactNativeMicrotasks (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:3169:48)
at anonymous (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:2983:43)
at _guard (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:3152:15)
at flushedQueue (http://192.168.11.210:8081/node_modules/expo-router/entry.bundle//&platform…1&transform.routerRoot=app&unstable_transformProfile=hermes-stable:2982:21)
anonymous @ C:\Project\ArabicLea…ills\console.js:589
reactConsoleErrorHandler @ C:\Project\ArabicLea…tionsManager.js:168
anonymous @ C:\Project\ArabicLea…eveloperTools.js:67
registerError @ C:\Project\ArabicLea…ogBox\LogBox.js:198
anonymous @ C:\Project\ArabicLea…LogBox\LogBox.js:68
overrideMethod @ C:\Project\ArabicLea…st\backend.js:14284
?anon_0 @ C:\Project\ArabicLea…[lessonId].tsx:245
asyncGeneratorStep @ C:\Project\ArabicLea…yncToGenerator.js:3
_throw @ C:\Project\ArabicLea…ncToGenerator.js:20
anonymous @ C:\Project\ArabicLea…ers\JSTimers.js:247
_callTimer @ C:\Project\ArabicLea…ers\JSTimers.js:111
_callReactNativeMicrotasksPass @ C:\Project\ArabicLea…ers\JSTimers.js:161
callReactNativeMicrotasks @ C:\Project\ArabicLea…ers\JSTimers.js:415
__callReactNativeMicrotasks @ C:\Project\ArabicLea…MessageQueue.js:393
anonymous @ C:\Project\ArabicLea…MessageQueue.js:132
__guard @ C:\Project\ArabicLea…MessageQueue.js:368
flushedQueue @ C:\Project\ArabicLea…MessageQueue.js:131
invokeCallbackAndReturnFlushedQueue @ C:\Project\ArabicLea…MessageQueue.js:127

Additional information:
"expo": "~52.0.35",
"expo-dev-client": "~5.0.12",

To reproduce

import * as ort from 'onnxruntime-react-native';
import { Asset } from 'expo-asset';

async function runInference(audioUri: string) {
try {
// Load model
const modelAsset = await Asset.loadAsync(require('./whisper-base-cpu-int8.onnx'));
const session = await InferenceSession.create(modelAsset[0].localUri);

const inputTensor = await processAudioForWhisper(audioUri);

// Example feeds
baseInputs = {
max_length: new ort.Tensor("int32", new Int32Array([200]), [1]),
min_length: new ort.Tensor("int32", new Int32Array([1]), [1]),
num_beams: new ort.Tensor("int32", new Int32Array([1]), [1]),
num_return_sequences: new ort.Tensor("int32", new Int32Array([1]), [1]),
length_penalty: new ort.Tensor("float32", new Float32Array([1.0]), [1]),
repetition_penalty: new ort.Tensor("float32", new Float32Array([1.0]), [1]),
decoder_input_ids: new ort.Tensor("int32", new Int32Array([50256]), [1, 1]),
};

const feeds: Record<string, ort.Tensor> = {
...baseInputs,
audio_pcm: inputTensor
};

// Run inference
const fetches = await session.run(feeds);
const output = fetches["str"];
console.log("Output:", { type: output.type, dims: output.dims, data: output.data });
// Attempting to access output.data[0] triggers the exception
} catch (error) {
console.error("Inference problem : ", error);
}
}
runInference();

Code for audio processing:
import * as FileSystem from 'expo-file-system';
import * as ort from 'onnxruntime-react-native';

interface AudioProcessingConfig {
sampleRate: number;
maxAudioLengthSeconds: number;
}

class AudioProcessor {
private static readonly DEFAULT_CONFIG: AudioProcessingConfig = {
sampleRate: 16000,
maxAudioLengthSeconds: 30
};

private config: AudioProcessingConfig;

constructor(config: Partial = {}) {
this.config = { ...AudioProcessor.DEFAULT_CONFIG, ...config };
}

async processAudioFile(fileUri: string): Promise<ort.Tensor> {
try {
// Read the PCM file as binary
const base64Data = await FileSystem.readAsStringAsync(fileUri, {
encoding: FileSystem.EncodingType.Base64
});
const audioBuffer = Buffer.from(base64Data, "base64"); // Convert to binary buffer

const floatArray = new Float32Array(audioBuffer.length / 4);
for (let i = 0; i < floatArray.length; i++) {
floatArray[i] = audioBuffer.readFloatLE(i * 4);
}
// Create audio tensor
return new ort.Tensor(
floatArray,
[1, floatArray.length]
);

} catch (error) {
  console.error('Error processing audio file:', error);
  throw error;
}

}
export async function processAudioForWhisper(fileUri: string): Promise<ort.Tensor> {
const processor = new AudioProcessor();
return processor.processAudioFile(fileUri);
}

Urgency

Please I’d be grateful if this could be addressed soon, as it’s currently preventing my project from moving forward.

Platform

React Native

OS Version

windows 11

ONNX Runtime Installation

Released Package

Compiler Version (if 'Built from Source')

No response

Package Name (if 'Released Package')

onnxruntime-react-native

ONNX Runtime Version or Commit ID

1.20.1

ONNX Runtime API

JavaScript

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[MAProcessing]

For audio recording options I use:
Audio.RecordingOptionsPresets.HIGH_QUALITY = {
isMeteringEnabled: true,
android: {
extension: '.m4a',
outputFormat: AndroidOutputFormat.MPEG_4,
audioEncoder: AndroidAudioEncoder.DEFAULT,
sampleRate: 16000,
numberOfChannels: 1,
bitRate: 16 * 16000 * 1,
},
ios: {
extension: '.wav',
outputFormat: IOSOutputFormat.MPEG4AAC,
audioQuality: IOSAudioQuality.MAX,
sampleRate: 16000,
numberOfChannels: 1,
bitRate: 16 * 16000 * 1,
linearPCMBitDepth: 16,
linearPCMIsBigEndian: false,
linearPCMIsFloat: false,
},
web: {
mimeType: 'audio/webm',
bitsPerSecond: 16 * 16000 * 1,
},
};

[microsoft-github-policy-service]

Applying stale label due to no activity in 30 days

[microsoft-github-policy-service]

Applying stale label due to no activity in 30 days