Skip to content

Commit 3290e1c

Browse files
csukuangfjcsukuangfj
authored
Support Fun-ASR-Nano-2512 (#2906)
This pull request significantly expands the capabilities of sherpa-onnx by integrating the Fun-ASR-Nano-2512 model. The changes encompass the entire pipeline, from model conversion and quantization using new Python scripts to specialized C++ inference logic that correctly interprets the Nano model's unique output structure and metadata. This ensures seamless support for this new, highly optimized ASR model, which boasts features like far-field high-noise recognition and multi-dialect/multi-language support.
1 parent 5f0049b commit 3290e1c

21 files changed

Lines changed: 910 additions & 57 deletions

‎.github/workflows/export-sense-voice-to-onnx.yaml‎

Lines changed: 98 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,10 +26,31 @@ jobs:
2626
with:
2727
python-version: ${{ matrix.python-version }}
2828

29+
- name: Install dependencies
30+
shell: bash
31+
run: |
32+
sudo apt-get install -y -qq sox libsox-fmt-mp3
33+
34+
- name: Install Python dependencies
35+
shell: bash
36+
run: |
37+
pip install \
38+
torch==2.0.0+cpu -f https://download.pytorch.org/whl/torch \
39+
onnx==1.17.0 \
40+
onnxruntime==1.17.1 \
41+
soundfile \
42+
kaldi-native-fbank \
43+
librosa
44+
45+
pip install "numpy<2"
46+
2947
- name: Download test_wavs
3048
shell: bash
3149
run: |
3250
sudo apt-get install -y -qq sox libsox-fmt-mp3
51+
52+
cd scripts/sense-voice
53+
3354
curl -SL -O https://huggingface.co/FunAudioLLM/SenseVoiceSmall/resolve/main/example/zh.mp3
3455
curl -SL -O https://huggingface.co/FunAudioLLM/SenseVoiceSmall/resolve/main/example/en.mp3
3556
curl -SL -O https://huggingface.co/FunAudioLLM/SenseVoiceSmall/resolve/main/example/ja.mp3
@@ -44,13 +65,90 @@ jobs:
4465
sox ko.mp3 -r 16k ko.wav
4566
sox yue.mp3 -r 16k yue.wav
4667
68+
69+
- name: Run
70+
shell: bash
71+
run: |
72+
cd scripts/sense-voice
73+
curl -SL -O https://huggingface.co/csukuangfj/funasr-nano-with-ctc/resolve/main/model.pt
74+
curl -SL -O https://huggingface.co/csukuangfj/funasr-nano-with-ctc/resolve/main/tokens.txt
75+
ls -lh
76+
./export_onnx_nano.py
77+
78+
ls -lh
79+
80+
d=sherpa-onnx-sense-voice-funasr-nano-2025-12-17
81+
d2=sherpa-onnx-sense-voice-funasr-nano-int8-2025-12-17
82+
mkdir -p $d $d2
83+
84+
cp README-nano.md $d/README.md
85+
cp README-nano.md $d2/README.md
86+
87+
mv model.onnx $d/
88+
mv model.int8.onnx $d2/
89+
90+
for m in $d $d2; do
91+
mkdir -p $m/test_wavs
92+
cp -v *.wav $m/test_wavs
93+
cp -v tokens.txt $m/
94+
95+
ls -lh $m
96+
97+
tar cjfv $m.tar.bz2 $m
98+
99+
ls -lh $m.tar.bz2
100+
mv $m.tar.bz2 ../../
101+
mv $m ../../
102+
done
103+
104+
- name: Publish v3 to huggingface
105+
if: true
106+
env:
107+
HF_TOKEN: ${{ secrets.HF_TOKEN }}
108+
uses: nick-fields/retry@v3
109+
with:
110+
max_attempts: 5
111+
timeout_seconds: 200
112+
shell: bash
113+
command: |
114+
git config --global user.email "csukuangfj@gmail.com"
115+
git config --global user.name "Fangjun Kuang"
116+
117+
names=(
118+
sherpa-onnx-sense-voice-funasr-nano-2025-12-17
119+
sherpa-onnx-sense-voice-funasr-nano-int8-2025-12-17
120+
)
121+
for d in ${names[@]}; do
122+
if [ ! -d $d ]; then
123+
echo "$d does not exist - skip it"
124+
continue;
125+
fi
126+
127+
export GIT_LFS_SKIP_SMUDGE=1
128+
export GIT_CLONE_PROTECTION_ACTIVE=false
129+
rm -rf huggingface
130+
git clone https://csukuangfj:$HF_TOKEN@huggingface.co/csukuangfj/$d huggingface
131+
cp -av $d/* ./huggingface
132+
cd huggingface
133+
git lfs track "*.onnx"
134+
git lfs track "*.wav"
135+
git status
136+
git add .
137+
git status
138+
git commit -m "add models"
139+
git push https://csukuangfj:$HF_TOKEN@huggingface.co/csukuangfj/$d main
140+
cd ..
141+
done
142+
47143
- name: Run
48144
shell: bash
145+
if: false
49146
run: |
50147
cd scripts/sense-voice
51148
./run.sh
52149
53150
- name: Publish to huggingface
151+
if: false
54152
env:
55153
HF_TOKEN: ${{ secrets.HF_TOKEN }}
56154
uses: nick-fields/retry@v3

‎scripts/sense-voice/README-nano.md‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Introduction
2+
3+
This directory contains models converted from
4+
https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512
5+
6+
## Core Features
7+
8+
> From https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512
9+
10+
- Far-field High-noise Recognition: Deeply optimized for far-distance sound pickup and high-noise scenarios (such as conference rooms, in-vehicle environments, industrial sites, etc.), improving recognition accuracy to 93%.
11+
12+
- Chinese Dialects and Regional Accents:
13+
14+
- Supports 7 major dialects: Wu, Cantonese, Min, Hakka, Gan, Xiang, Jin
15+
- Covers 26 regional accents: including Henan, Shaanxi, Hubei, Sichuan, Chongqing, Yunnan, Guizhou, Guangdong, Guangxi and more than 20 other regions
16+
17+
- Multi-language Free Speech: Supports recognition of 31 languages, with focused optimization on East and Southeast Asian languages, supporting free language switching and mixed recognition.
18+
- Music Background Lyric Recognition: Enhanced speech recognition performance under music background interference, supporting accurate recognition of lyric content in songs.
19+
20+
21+
22+
## 核心特性
23+
24+
> From https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512/blob/main/README_zh.md
25+
26+
- 远场高噪声识别： 针对远距离拾音及高噪声场景（如会议室、车载环境、工业现场等）进行深度优化，识别准确率提升至 **93%**。
27+
- 中文方言与地方口音：
28+
29+
- 支持 7 大方言：吴语、粤语、闽语、客家话、赣语、湘语、晋语
30+
- 覆盖 26 个地区口音：包括河南、陕西、湖北、四川、重庆、云南、贵州、广东、广西等 20 多个地区
31+
32+
- 多语言自由说： 支持 31 种语言识别，重点优化东亚与东南亚语种，支持语种自由切换和混合识别。
33+
- 音乐背景歌词识别： 强化在音乐背景干扰下的语音识别性能，支持对歌曲中歌词内容的精准识别。

‎scripts/sense-voice/adaptor.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
rknn/adaptor.py

‎scripts/sense-voice/export-onnx.py‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@
77
as a reference while writing this file.
88
99
Thanks to https://github.com/yuekaizhang for making the file public.
10+
11+
You should install FunASR before you run this file.
1012
"""
1113

1214
import os
@@ -120,7 +122,9 @@ def display_params(params):
120122

121123
@torch.no_grad()
122124
def main():
123-
model, params = SenseVoiceSmall.from_pretrained(model="iic/SenseVoiceSmall", device="cpu")
125+
model, params = SenseVoiceSmall.from_pretrained(
126+
model="iic/SenseVoiceSmall", device="cpu"
127+
)
124128
model.eval()
125129

126130
display_params(params)

‎scripts/sense-voice/export_onnx_nano.py‎

Lines changed: 112 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,112 @@
1+
#!/usr/bin/env python3
2+
# Copyright 2025 Xiaomi Corp. (authors: Fangjun Kuang)
3+
4+
import argparse
5+
import os
6+
from typing import Any, Dict
7+
8+
import onnx
9+
import torch
10+
from onnxruntime.quantization import QuantType, quantize_dynamic
11+
12+
from test_nano_torch import load_tokens, load_torch_model
13+
14+
15+
def get_args():
16+
parser = argparse.ArgumentParser(
17+
formatter_class=argparse.ArgumentDefaultsHelpFormatter
18+
)
19+
20+
parser.add_argument(
21+
"--opset-version",
22+
type=int,
23+
default=13,
24+
)
25+
return parser.parse_args()
26+
27+
28+
def add_meta_data(filename: str, meta_data: Dict[str, Any]):
29+
"""Add meta data to an ONNX model. It is changed in-place.
30+
31+
Args:
32+
filename:
33+
Filename of the ONNX model to be changed.
34+
meta_data:
35+
Key-value pairs.
36+
"""
37+
model = onnx.load(filename)
38+
while len(model.metadata_props):
39+
model.metadata_props.pop()
40+
41+
for key, value in meta_data.items():
42+
meta = model.metadata_props.add()
43+
meta.key = key
44+
meta.value = str(value)
45+
46+
onnx.save(model, filename)
47+
48+
49+
@torch.no_grad()
50+
def main():
51+
args = get_args()
52+
print(vars(args))
53+
id2tokens = load_tokens()
54+
55+
vocab_size = len(id2tokens)
56+
blank_id = vocab_size - 1
57+
58+
print("loading model")
59+
60+
model = load_torch_model()
61+
model.eval()
62+
63+
x = torch.randn(1, 30, 560, dtype=torch.float32)
64+
65+
opset_version = args.opset_version
66+
filename = "model.onnx"
67+
torch.onnx.export(
68+
model,
69+
x,
70+
filename,
71+
opset_version=opset_version,
72+
input_names=["x"],
73+
output_names=["logits"],
74+
dynamic_axes={
75+
"x": {1: "T"},
76+
},
77+
)
78+
79+
model_author = "FunAudioLLM"
80+
comment = os.environ.get("comment", "FunAudioLLM/Fun-ASR-Nano-2512")
81+
url = "https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512"
82+
83+
meta_data = {
84+
"lfr_window_size": 7,
85+
"lfr_window_shift": 6,
86+
"normalize_samples": 0, # input should be in the range [-32768, 32767]
87+
"model_type": "sense_voice_ctc",
88+
"version": "1",
89+
"model_author": model_author,
90+
"maintainer": "k2-fsa",
91+
"vocab_size": vocab_size,
92+
"blank_id": blank_id,
93+
"comment": comment,
94+
"url": url,
95+
}
96+
add_meta_data(filename=filename, meta_data=meta_data)
97+
98+
filename_int8 = "model.int8.onnx"
99+
quantize_dynamic(
100+
model_input=filename,
101+
model_output=filename_int8,
102+
op_types_to_quantize=["MatMul"],
103+
# Note that we have to use QUInt8 here.
104+
#
105+
# When QInt8 is used, C++ onnxruntime produces incorrect results
106+
weight_type=QuantType.QUInt8,
107+
)
108+
109+
110+
if __name__ == "__main__":
111+
torch.manual_seed(20251217)
112+
main()

‎scripts/sense-voice/nano.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
rknn/nano.py

0 commit comments

Comments
 (0)