Skip to content

Commit b82d9c6

Browse files
Wasser1462Wasser1462
authored
FunASR-nano: switch to unified KV-cache LLM (#2995)
This PR updates FunASR-nano inference from the prefill+decode dual-model pipeline to a single unified KV-cache model. ## Summary Previously, FunASR-nano required two separate ONNX models: - `llm_prefill.onnx` - `llm_decode.onnx` This PR switches to a single model: - `llm.onnx` The new pipeline uses a static KV cache + KV-delta incremental update mechanism, and relies on `cache_position` to differentiate prefill vs. decode steps. This significantly simplifies model/session management and reduces deployment complexity. ## Key changes - **Single LLM session / single model file**: `llm.onnx` replaces `llm_prefill.onnx` + `llm_decode.onnx`. - **Unified KV-cache implementation**: - static KV cache layout - KV-delta update for decode - `cache_position` distinguishes prefill vs. decode behavior - **Config changes (breaking)**: - `funasr_nano.llm_prefill` and `funasr_nano.llm_decode` are deprecated/removed - use only `funasr_nano.llm` - **Not backward compatible**: - users must re-export models in KV-delta/unified-KV format - **Trade-off**: slightly slower, but lower VRAM duplication
1 parent c305335 commit b82d9c6

17 files changed

Lines changed: 1038 additions & 584 deletions

‎cxx-api-examples/funasr-nano-cxx-api.cc‎

Lines changed: 12 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -9,8 +9,7 @@
99
// Example usage:
1010
// ./bin/funasr-nano-cxx-api \
1111
// --funasr-nano-encoder-adaptor=/path/to/encoder_adaptor.onnx \
12-
// --funasr-nano-llm-prefill=/path/to/llm_prefill.onnx \
13-
// --funasr-nano-llm-decode=/path/to/llm_decode.onnx \
12+
// --funasr-nano-llm=/path/to/llm.onnx \
1413
// --funasr-nano-embedding=/path/to/embedding.onnx \
1514
// --funasr-nano-tokenizer=/path/to/Qwen3-0.6B \
1615
// /path/to/audio.wav
@@ -33,36 +32,33 @@ FunASR-nano speech recognition example using sherpa-onnx C++ API.
3332
Usage:
3433
./bin/funasr-nano-cxx-api \
3534
--funasr-nano-encoder-adaptor=/path/to/encoder_adaptor.onnx \
36-
--funasr-nano-llm-prefill=/path/to/llm_prefill.onnx \
37-
--funasr-nano-llm-decode=/path/to/llm_decode.onnx \
35+
--funasr-nano-llm=/path/to/llm.onnx \
3836
--funasr-nano-tokenizer=/path/to/Qwen3-0.6B \
3937
--funasr-nano-embedding=/path/to/embedding.onnx \
4038
[--funasr-nano-user-prompt="语音转写："] \
4139
[--funasr-nano-max-new-tokens=512] \
42-
[--funasr-nano-temperature=0.3] \
40+
[--funasr-nano-temperature=1e-6] \
4341
[--funasr-nano-top-p=0.8] \
4442
/path/to/audio.wav
4543
4644
Required arguments:
4745
--funasr-nano-encoder-adaptor: Path to encoder_adaptor.onnx
48-
--funasr-nano-llm-prefill: Path to llm_prefill.onnx
49-
--funasr-nano-llm-decode: Path to llm_decode.onnx
46+
--funasr-nano-llm: Path to llm.onnx (unified KV cache model)
5047
--funasr-nano-tokenizer: Path to tokenizer directory (e.g., Qwen3-0.6B)
5148
--funasr-nano-embedding: Path to embedding.onnx
5249
5350
Optional arguments:
5451
--funasr-nano-user-prompt: User prompt template (default: "语音转写：")
5552
--funasr-nano-max-new-tokens: Maximum tokens to generate (default: 512)
56-
--funasr-nano-temperature: Sampling temperature (default: 0.3)
53+
--funasr-nano-temperature: Sampling temperature (default: 1e-6)
5754
--funasr-nano-top-p: Top-p sampling threshold (default: 0.8)
5855
--num-threads: Number of threads (default: 2)
5956
--provider: cpu (default) or cuda
6057
6158
Example:
6259
./bin/funasr-nano-cxx-api \
6360
--funasr-nano-encoder-adaptor=./models/encoder_adaptor.onnx \
64-
--funasr-nano-llm-prefill=./models/llm_prefill.onnx \
65-
--funasr-nano-llm-decode=./models/llm_decode.onnx \
61+
--funasr-nano-llm=./models/llm.onnx \
6662
--funasr-nano-tokenizer=./models/Qwen3-0.6B \
6763
--funasr-nano-embedding=./models/embedding.onnx \
6864
./test.wav
@@ -80,8 +76,7 @@ Optional arguments:
8076

8177
// Parse command line arguments
8278
const char kEncoderAdaptor[] = "--funasr-nano-encoder-adaptor=";
83-
const char kLlmPrefill[] = "--funasr-nano-llm-prefill=";
84-
const char kLlmDecode[] = "--funasr-nano-llm-decode=";
79+
const char kLlm[] = "--funasr-nano-llm=";
8580
const char kEmbedding[] = "--funasr-nano-embedding=";
8681
const char kTokenizer[] = "--funasr-nano-tokenizer=";
8782
const char kUserPrompt[] = "--funasr-nano-user-prompt=";
@@ -96,12 +91,9 @@ Optional arguments:
9691
if (arg.find(kEncoderAdaptor) == 0) {
9792
config.model_config.funasr_nano.encoder_adaptor =
9893
arg.substr(sizeof(kEncoderAdaptor) - 1);
99-
} else if (arg.find(kLlmPrefill) == 0) {
100-
config.model_config.funasr_nano.llm_prefill =
101-
arg.substr(sizeof(kLlmPrefill) - 1);
102-
} else if (arg.find(kLlmDecode) == 0) {
103-
config.model_config.funasr_nano.llm_decode =
104-
arg.substr(sizeof(kLlmDecode) - 1);
94+
} else if (arg.find(kLlm) == 0) {
95+
config.model_config.funasr_nano.llm =
96+
arg.substr(sizeof(kLlm) - 1);
10597
} else if (arg.find(kEmbedding) == 0) {
10698
config.model_config.funasr_nano.embedding =
10799
arg.substr(sizeof(kEmbedding) - 1);
@@ -132,10 +124,8 @@ Optional arguments:
132124
std::cout << "Loading model...\n";
133125
std::cout << " encoder_adaptor: "
134126
<< config.model_config.funasr_nano.encoder_adaptor << "\n";
135-
std::cout << " llm_prefill: "
136-
<< config.model_config.funasr_nano.llm_prefill << "\n";
137-
std::cout << " llm_decode: "
138-
<< config.model_config.funasr_nano.llm_decode << "\n";
127+
std::cout << " llm: "
128+
<< config.model_config.funasr_nano.llm << "\n";
139129
std::cout << " tokenizer: " << config.model_config.funasr_nano.tokenizer
140130
<< "\n";
141131
std::cout << " embedding: "

‎python-api-examples/offline-funasr-nano-decode-files.py‎

Lines changed: 6 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,7 @@
1010
Usage:
1111
python offline-funasr-nano-decode-files.py \
1212
--encoder-adaptor=/path/to/encoder_adaptor.onnx \
13-
--llm-prefill=/path/to/llm_prefill.onnx \
14-
--llm-decode=/path/to/llm_decode.onnx \
13+
--llm=/path/to/llm.onnx \
1514
--tokenizer=/path/to/Qwen3-0.6B \
1615
--embedding=/path/to/embedding.onnx \
1716
[--num-threads=4] \
@@ -46,17 +45,10 @@ def get_args():
4645
)
4746

4847
parser.add_argument(
49-
"--llm-prefill",
48+
"--llm",
5049
type=str,
5150
required=True,
52-
help="Path to llm_prefill.onnx (KV cache mode)",
53-
)
54-
55-
parser.add_argument(
56-
"--llm-decode",
57-
type=str,
58-
required=True,
59-
help="Path to llm_decode.onnx (KV cache mode)",
51+
help="Path to llm.onnx (unified KV cache model)",
6052
)
6153

6254
parser.add_argument(
@@ -83,7 +75,7 @@ def get_args():
8375
parser.add_argument(
8476
"--user-prompt",
8577
type=str,
86-
default="Transcription:",
78+
default="语音转写:",
8779
help="User prompt template for FunASR-nano",
8880
)
8981

@@ -97,7 +89,7 @@ def get_args():
9789
parser.add_argument(
9890
"--temperature",
9991
type=float,
100-
default=0.3,
92+
default=1e-6,
10193
help="Sampling temperature",
10294
)
10395

@@ -151,8 +143,7 @@ def get_args():
151143
def create_recognizer(args) -> sherpa_onnx.OfflineRecognizer:
152144
return sherpa_onnx.OfflineRecognizer.from_funasr_nano(
153145
encoder_adaptor=args.encoder_adaptor,
154-
llm_prefill=args.llm_prefill,
155-
llm_decode=args.llm_decode,
146+
llm=args.llm,
156147
embedding=args.embedding,
157148
tokenizer=args.tokenizer,
158149
num_threads=args.num_threads,

‎scripts/go/sherpa_onnx.go‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -454,8 +454,7 @@ type OfflineFireRedAsrModelConfig struct {
454454

455455
type OfflineFunASRNanoModelConfig struct {
456456
EncoderAdaptor string
457-
LlmPreFill string
458-
LlmDecoder string
457+
Llm string
459458
Embedding string
460459
Tokenizer string
461460
SystemPrompt string
@@ -598,8 +597,7 @@ func newCOfflineRecognizerConfig(config *OfflineRecognizerConfig) *C.struct_Sher
598597
c.model_config.fire_red_asr.decoder = C.CString(config.ModelConfig.FireRedAsr.Decoder)
599598

600599
c.model_config.funasr_nano.encoder_adaptor = C.CString(config.ModelConfig.FunASRNano.EncoderAdaptor)
601-
c.model_config.funasr_nano.llm_prefill = C.CString(config.ModelConfig.FunASRNano.LlmPreFill)
602-
c.model_config.funasr_nano.llm_decode = C.CString(config.ModelConfig.FunASRNano.LlmDecoder)
600+
c.model_config.funasr_nano.llm = C.CString(config.ModelConfig.FunASRNano.Llm)
603601
c.model_config.funasr_nano.embedding = C.CString(config.ModelConfig.FunASRNano.Embedding)
604602
c.model_config.funasr_nano.tokenizer = C.CString(config.ModelConfig.FunASRNano.Tokenizer)
605603
c.model_config.funasr_nano.system_prompt = C.CString(config.ModelConfig.FunASRNano.SystemPrompt)
@@ -679,8 +677,7 @@ func freeCOfflineRecognizerConfig(c *C.struct_SherpaOnnxOfflineRecognizerConfig)
679677
&c.model_config.fire_red_asr.encoder,
680678
&c.model_config.fire_red_asr.decoder,
681679
&c.model_config.funasr_nano.encoder_adaptor,
682-
&c.model_config.funasr_nano.llm_prefill,
683-
&c.model_config.funasr_nano.llm_decode,
680+
&c.model_config.funasr_nano.llm,
684681
&c.model_config.funasr_nano.embedding,
685682
&c.model_config.funasr_nano.tokenizer,
686683
&c.model_config.funasr_nano.system_prompt,

‎sherpa-onnx/c-api/c-api.cc‎

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -513,10 +513,8 @@ static sherpa_onnx::OfflineRecognizerConfig GetOfflineRecognizerConfig(
513513

514514
recognizer_config.model_config.funasr_nano.encoder_adaptor =
515515
SHERPA_ONNX_OR(config->model_config.funasr_nano.encoder_adaptor, "");
516-
recognizer_config.model_config.funasr_nano.llm_prefill =
517-
SHERPA_ONNX_OR(config->model_config.funasr_nano.llm_prefill, "");
518-
recognizer_config.model_config.funasr_nano.llm_decode =
519-
SHERPA_ONNX_OR(config->model_config.funasr_nano.llm_decode, "");
516+
recognizer_config.model_config.funasr_nano.llm =
517+
SHERPA_ONNX_OR(config->model_config.funasr_nano.llm, "");
520518
recognizer_config.model_config.funasr_nano.embedding =
521519
SHERPA_ONNX_OR(config->model_config.funasr_nano.embedding, "");
522520
recognizer_config.model_config.funasr_nano.tokenizer =
@@ -530,7 +528,7 @@ static sherpa_onnx::OfflineRecognizerConfig GetOfflineRecognizerConfig(
530528
recognizer_config.model_config.funasr_nano.max_new_tokens =
531529
SHERPA_ONNX_OR(config->model_config.funasr_nano.max_new_tokens, 512);
532530
recognizer_config.model_config.funasr_nano.temperature =
533-
SHERPA_ONNX_OR(config->model_config.funasr_nano.temperature, 0.3f);
531+
SHERPA_ONNX_OR(config->model_config.funasr_nano.temperature, 1e-6f);
534532
recognizer_config.model_config.funasr_nano.top_p =
535533
SHERPA_ONNX_OR(config->model_config.funasr_nano.top_p, 0.8f);
536534
recognizer_config.model_config.funasr_nano.seed =

‎sherpa-onnx/c-api/c-api.h‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -486,8 +486,7 @@ SHERPA_ONNX_API typedef struct SherpaOnnxOfflineOmnilingualAsrCtcModelConfig {
486486

487487
SHERPA_ONNX_API typedef struct SherpaOnnxOfflineFunASRNanoModelConfig {
488488
const char *encoder_adaptor;
489-
const char *llm_prefill;
490-
const char *llm_decode;
489+
const char *llm;
491490
const char *embedding;
492491
const char *tokenizer;
493492
const char *system_prompt;

‎sherpa-onnx/c-api/cxx-api.cc‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -274,10 +274,8 @@ static SherpaOnnxOfflineRecognizerConfig Convert(
274274

275275
c.model_config.funasr_nano.encoder_adaptor =
276276
config.model_config.funasr_nano.encoder_adaptor.c_str();
277-
c.model_config.funasr_nano.llm_prefill =
278-
config.model_config.funasr_nano.llm_prefill.c_str();
279-
c.model_config.funasr_nano.llm_decode =
280-
config.model_config.funasr_nano.llm_decode.c_str();
277+
c.model_config.funasr_nano.llm =
278+
config.model_config.funasr_nano.llm.c_str();
281279
c.model_config.funasr_nano.embedding =
282280
config.model_config.funasr_nano.embedding.c_str();
283281
c.model_config.funasr_nano.tokenizer =

‎sherpa-onnx/c-api/cxx-api.h‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -285,14 +285,13 @@ struct SHERPA_ONNX_API OfflineMoonshineModelConfig {
285285

286286
struct SHERPA_ONNX_API OfflineFunASRNanoModelConfig {
287287
std::string encoder_adaptor;
288-
std::string llm_prefill;
289-
std::string llm_decode;
288+
std::string llm;
290289
std::string embedding;
291290
std::string tokenizer;
292291
std::string system_prompt = "You are a helpful assistant.";
293292
std::string user_prompt = "语音转写：";
294293
int32_t max_new_tokens = 512;
295-
float temperature = 0.3f;
294+
float temperature = 1e-6f;
296295
float top_p = 0.8f;
297296
int32_t seed = 42;
298297
};

‎sherpa-onnx/csrc/offline-funasr-nano-model-config.cc‎

Lines changed: 7 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,8 @@ void OfflineFunASRNanoModelConfig::Register(ParseOptions *po) {
1515
po->Register("funasr-nano-encoder-adaptor", &encoder_adaptor,
1616
"Path to encoder_adaptor.onnx for FunASR-nano");
1717

18-
po->Register("funasr-nano-llm-prefill", &llm_prefill,
19-
"Path to llm_prefill.onnx for FunASR-nano (KV cache mode)");
20-
21-
po->Register("funasr-nano-llm-decode", &llm_decode,
22-
"Path to llm_decode.onnx for FunASR-nano (KV cache mode)");
18+
po->Register("funasr-nano-llm", &llm,
19+
"Path to llm.onnx for FunASR-nano (KV cache mode)");
2320

2421
po->Register("funasr-nano-embedding", &embedding,
2522
"Path to embedding.onnx for FunASR-nano");
@@ -58,18 +55,13 @@ bool OfflineFunASRNanoModelConfig::Validate() const {
5855
return false;
5956
}
6057

61-
// KV cache mode (prefill + decode) is required
62-
if (llm_prefill.empty() || llm_decode.empty()) {
63-
SHERPA_ONNX_LOGE("Both --funasr-nano-llm-prefill and --funasr-nano-llm-decode are required");
58+
if (llm.empty()) {
59+
SHERPA_ONNX_LOGE("--funasr-nano-llm is required");
6460
return false;
6561
}
6662

67-
if (!FileExists(llm_prefill)) {
68-
SHERPA_ONNX_LOGE("--funasr-nano-llm-prefill: '%s' does not exist", llm_prefill.c_str());
69-
return false;
70-
}
71-
if (!FileExists(llm_decode)) {
72-
SHERPA_ONNX_LOGE("--funasr-nano-llm-decode: '%s' does not exist", llm_decode.c_str());
63+
if (!FileExists(llm)) {
64+
SHERPA_ONNX_LOGE("--funasr-nano-llm: '%s' does not exist", llm.c_str());
7365
return false;
7466
}
7567

@@ -121,8 +113,7 @@ std::string OfflineFunASRNanoModelConfig::ToString() const {
121113

122114
os << "OfflineFunASRNanoModelConfig(";
123115
os << "encoder_adaptor=\"" << encoder_adaptor << "\", ";
124-
os << "llm_prefill=\"" << llm_prefill << "\", ";
125-
os << "llm_decode=\"" << llm_decode << "\", ";
116+
os << "llm=\"" << llm << "\", ";
126117
os << "embedding=\"" << embedding << "\", ";
127118
os << "tokenizer=\"" << tokenizer << "\", ";
128119
os << "system_prompt=\"" << system_prompt << "\", ";

‎sherpa-onnx/csrc/offline-funasr-nano-model-config.h‎

Lines changed: 3 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,8 @@ struct OfflineFunASRNanoModelConfig {
1515
// Path to encoder_adaptor.onnx
1616
std::string encoder_adaptor;
1717

18-
// Path to llm_prefill.onnx (KV cache prefill)
19-
std::string llm_prefill;
20-
21-
// Path to llm_decode.onnx (KV cache decode)
22-
std::string llm_decode;
18+
// Path to llm.onnx (KV cache model)
19+
std::string llm;
2320

2421
// Path to embedding.onnx
2522
std::string embedding;
@@ -37,7 +34,7 @@ struct OfflineFunASRNanoModelConfig {
3734
int32_t max_new_tokens = 512;
3835

3936
// Sampling temperature
40-
float temperature = 0.3f;
37+
float temperature = 1e-6f;
4138

4239
// Top-p (nucleus) sampling threshold
4340
float top_p = 0.8f;

0 commit comments

Comments
 (0)