Export Whisper to RK NPU (#2983)

Author: csukuangfj

.gitignore

@@ -166,3 +166,5 @@ spacemit-toolchain*
 sherpa-onnx-qnn-*
 matcha-icefall-*
 sherpa-onnx-medasr-ctc-en-int8-2025-12-25
+*.raw
+*-input-list.txt

README.md

@@ -424,10 +424,6 @@ It uses Swift for iOS and Java for Android.
 Flet ASR/STT component based on sherpa-onnx.
 Example [a chat box agent](https://github.com/SamYuan1990/i18n-agent-action)
 
-### [elderly-companion](https://github.com/SearocIsMe/elderly-companion)
-
-It uses sherpa-onnx's Python API for real-time speech recognition in ROS2 with RK NPU.
-
 ### [achatbot-go](https://github.com/ai-bot-pro/achatbot-go)
 
 a multimodal chatbot based on go with sherpa-onnx's speech lib api.

scripts/whisper/export-onnx.py

@@ -32,9 +32,6 @@
     TextDecoder,
 )
 
-torch.set_num_threads(1)
-torch.set_num_interop_threads(1)
-
 
 def get_args():
     parser = argparse.ArgumentParser()
@@ -321,7 +318,7 @@ def main():
     print(args)
     print(name)
 
-    opset_version = 13
+    opset_version = 17
 
     if name == "distil-medium.en":
         filename = "./distil-medium-en-original-model.bin"
@@ -640,4 +637,12 @@ def main():
 
 
 if __name__ == "__main__":
-    main()
+    torch.set_num_threads(1)
+    torch.set_num_interop_threads(1)
+    # To fix
+    # TypeError: scaled_dot_product_attention(): argument 'is_causal' must be bool, not Tensor
+    # See also https://github.com/k2-fsa/sherpa-onnx/issues/1764
+    from whisper.model import disable_sdpa
+
+    with disable_sdpa():
+        main()

scripts/whisper/export_onnx.py

@@ -0,0 +1 @@
+export-onnx.py

scripts/whisper/model-info.md

@@ -0,0 +1,81 @@
+# tiny/tiny.en
+```
+ModelDimensions(
+    n_mels=80,
+    n_audio_ctx=1500,
+    n_audio_state=384,
+    n_audio_head=6,
+    n_audio_layer=4,
+    n_vocab=51865,
+    n_text_ctx=448,
+    n_text_state=384,
+    n_text_head=6,
+    n_text_layer=4
+)
+```
+
+# base/base.en
+```
+ModelDimensions(
+    n_mels=80,
+    n_audio_ctx=1500,
+    n_audio_state=512,
+    n_audio_head=8,
+    n_audio_layer=6,
+    n_vocab=51865,
+    n_text_ctx=448,
+    n_text_state=512,
+    n_text_head=8,
+    n_text_layer=6
+)
+```
+
+# small/small.en
+```
+ModelDimensions(
+    n_mels=80,
+    n_audio_ctx=1500,
+    n_audio_state=768,
+    n_audio_head=12,
+    n_audio_layer=12,
+    n_vocab=51865,
+    n_text_ctx=448,
+    n_text_state=768,
+    n_text_head=12,
+    n_text_layer=12
+)
+```
+
+
+# medium/medium.en
+```
+ModelDimensions(
+    n_mels=80,
+    n_audio_ctx=1500,
+    n_audio_state=1024,
+    n_audio_head=16,
+    n_audio_layer=24,
+    n_vocab=51865,
+    n_text_ctx=448,
+    n_text_state=1024,
+    n_text_head=16,
+    n_text_layer=24
+)
+```
+
+# large
+```
+ModelDimensions(
+    n_mels=80,
+    n_audio_ctx=1500,
+    n_audio_state=1280,
+    n_audio_head=20,
+    n_audio_layer=32,
+    n_vocab=51865,
+    n_text_ctx=448,
+    n_text_state=1280,
+    n_text_head=20,
+    n_text_layer=32
+)
+```
+

scripts/whisper/rknn/README.md

@@ -0,0 +1,48 @@
+# Usage
+
+You can find pre-exported rknn models for rk3588 at
+
+https://modelscope.cn/models/csukuangfj/2026-01-05-rknn/files
+
+
+# Download test wave
+
+```
+wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/en.wav
+wget https://github.com/k2-fsa/sherpa-onnx/releases/download/asr-models/en-16k.wav
+```
+
+## Export to onnx
+
+```
+./export_onnx.py --model tiny.en
+```
+
+## Test onnx
+
+```
+./test_onnx.py --model tiny.en
+```
+
+## Export to rknn
+
+```
+python3 ./export_rknn.py --target-platform rk3588  --in-model ./tiny.en-encoder.onnx --out-model ./tiny.en-encoder.rknn
+
+python3 ./export_rknn.py --target-platform rk3588  --in-model ./tiny.en-decoder.onnx --out-model ./tiny.en-decoder.rknn
+```
+
+```
+ls -lh tiny.en-*.rknn
+
+-rw-r--r-- 1 kuangfangjun root 95M Jan  5 16:16 tiny.en-decoder.rknn
+-rw-r--r-- 1 kuangfangjun root 22M Jan  5 16:15 tiny.en-encoder.rknn
+```
+
+## Run it on your rk3588 board
+
+```
+wget https://huggingface.co/csukuangfj/sherpa-onnx-whisper-tiny.en/resolve/main/tiny.en-tokens.txt
+
+./test_on_rk3588_board.py  --encoder ./tiny.en-encoder.rknn --decoder ./tiny.en-decoder.rknn --tokens ./tiny.en-tokens.txt --wav ./en-16k.wav
+```