Add C++ runtime for Parakeet TDT models with QNN#3720
Conversation
dosubot
Bot
added
the
size:XXL
📝 WalkthroughWalkthroughThis PR adds Parakeet TDT model support for QNN backends, including a new offline model implementation with encoder/decoder/joiner pipelines, a corresponding recognizer with greedy-search decoding, float16 tensor I/O support in the QNN runtime, and factory wiring. Minor unrelated cleanups touch the NeMo CTC model file. ChangesParakeet TDT QNN Implementation
Estimated code review effort: 4 (Complex) | ~60 minutes NeMo CTC Model Minor Fixes
Estimated code review effort: 1 (Trivial) | ~5 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant OfflineRecognizerImpl
participant OfflineRecognizerParakeetTdtQnnImpl
participant OfflineParakeetTdtModelQnn
Client->>OfflineRecognizerImpl: Create(config)
OfflineRecognizerImpl->>OfflineRecognizerImpl: IsQnnTransducerArtifact + model_type check
OfflineRecognizerImpl-->>Client: OfflineRecognizerParakeetTdtQnnImpl
Client->>OfflineRecognizerParakeetTdtQnnImpl: DecodeStreams(streams)
OfflineRecognizerParakeetTdtQnnImpl->>OfflineParakeetTdtModelQnn: RunEncoder(frames)
OfflineRecognizerParakeetTdtQnnImpl->>OfflineRecognizerParakeetTdtQnnImpl: GreedySearch(encoder_out)
loop per decoded token
OfflineRecognizerParakeetTdtQnnImpl->>OfflineParakeetTdtModelQnn: RunDecoder / RunJoiner
OfflineParakeetTdtModelQnn-->>OfflineRecognizerParakeetTdtQnnImpl: token, timestamp, duration
end
OfflineRecognizerParakeetTdtQnnImpl-->>Client: OfflineRecognitionResult
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for the Parakeet TDT model in QNN, including the implementation of the offline model, its corresponding recognizer, and integration into the build system and recognizer factory. Additionally, it adds float16 tensor data conversion utilities to the QNN backend. The review feedback highlights several robustness improvements, specifically recommending validation checks to prevent out-of-bounds reads when accessing joiner output logits, ensuring model filenames are provided when context binaries are absent, and verifying that the decoder states vector contains the expected number of elements before access.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| float *token_logits = logits.data(); | ||
| int32_t output_size = static_cast<int32_t>(logits.size()); | ||
| int32_t num_durations = output_size - vocab_size; | ||
| const float *duration_logits = logits.data() + vocab_size; |
There was a problem hiding this comment.
To prevent potential out-of-bounds reads and crashes when a mismatched model or tokens.txt is provided, we should validate that the joiner output size is at least equal to vocab_size before accessing duration_logits.
float *token_logits = logits.data();
int32_t output_size = static_cast<int32_t>(logits.size());
if (output_size < vocab_size) {
SHERPA_ONNX_LOGE(
"Joiner output size %d is less than vocab_size %d. "
"Please check your model and tokens.txt",
output_size, vocab_size);
SHERPA_ONNX_EXIT(-1);
}
int32_t num_durations = output_size - vocab_size;
const float *duration_logits = logits.data() + vocab_size;| if (!context_binaries_.empty() && context_binaries_.size() != 3) { | ||
| SHERPA_ONNX_LOGE( | ||
| "There should be 3 files for offline parakeet TDT context binary. " | ||
| "Actual: %d. '%s'", | ||
| static_cast<int32_t>(context_binaries_.size()), | ||
| config_.transducer.qnn_config.context_binary.c_str()); | ||
| SHERPA_ONNX_EXIT(-1); | ||
| } |
There was a problem hiding this comment.
If context_binaries_ is empty, we must ensure that encoder_filename, decoder_filename, and joiner_filename are not empty. Otherwise, the model will fail to load with a generic error. Adding a validation check here improves usability and robustness.
if (!context_binaries_.empty() && context_binaries_.size() != 3) {
SHERPA_ONNX_LOGE(
"There should be 3 files for offline parakeet TDT context binary. "
"Actual: %d. '%s'",
static_cast<int32_t>(context_binaries_.size()),
config_.transducer.qnn_config.context_binary.c_str());
SHERPA_ONNX_EXIT(-1);
}
if (context_binaries_.empty()) {
if (config_.transducer.encoder_filename.empty() ||
config_.transducer.decoder_filename.empty() ||
config_.transducer.joiner_filename.empty()) {
SHERPA_ONNX_LOGE(
"Please provide encoder_filename, decoder_filename, and "
"joiner_filename when context_binary is not provided.");
SHERPA_ONNX_EXIT(-1);
}
}| std::lock_guard<std::mutex> lock(mutex_); | ||
| decoder_->SetInputTensorData(decoder_input_y_name_, &token, 1); | ||
| decoder_->SetInputTensorData(decoder_input_h_name_, states[0].data(), | ||
| static_cast<int32_t>(states[0].size())); | ||
| decoder_->SetInputTensorData(decoder_input_c_name_, states[1].data(), | ||
| static_cast<int32_t>(states[1].size())); |
There was a problem hiding this comment.
To prevent potential out-of-bounds access or crashes, we should validate that states contains at least 2 elements (representing the hidden state h and cell state c) before accessing states[0] and states[1].
if (states.size() < 2) {
SHERPA_ONNX_LOGE("states should contain at least 2 elements (h and c).");
SHERPA_ONNX_EXIT(-1);
}
std::lock_guard<std::mutex> lock(mutex_);
decoder_->SetInputTensorData(decoder_input_y_name_, &token, 1);
decoder_->SetInputTensorData(decoder_input_h_name_, states[0].data(),
static_cast<int32_t>(states[0].size()));
decoder_->SetInputTensorData(decoder_input_c_name_, states[1].data(),
static_cast<int32_t>(states[1].size()));![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
sherpa-onnx/csrc/qnn/qnn-model.cc (1)
121-122: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winAvoid the full duplicate copy of the context binary.
ReadFilealready returns astd::vector<char>, andbufferis only consumed asstatic_cast<void *>(buffer.data())+buffer.size()(Lines 135, 189). Copying it into astd::vector<uint8_t>doubles peak memory while the binary is held — noticeable for large Parakeet TDT context binaries on constrained NPU devices. Use the char buffer directly.♻️ Proposed change
- auto char_buffer = ReadFile(binary_context_file); - std::vector<uint8_t> buffer(char_buffer.begin(), char_buffer.end()); + auto buffer = ReadFile(binary_context_file);
buffer.data()(nowchar *) still casts tovoid *at Lines 135 and 189 without change.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@sherpa-onnx/csrc/qnn/qnn-model.cc` around lines 121 - 122, The context binary is being duplicated unnecessarily in qnn-model.cc; use the existing ReadFile result directly instead of copying into a std::vector<uint8_t>. Update the code around the binary_context_file read path in qnn-model.cc so the existing char buffer is passed through to the later buffer.data()/buffer.size() consumers, and keep the handling consistent in the code paths that use the buffer for QNN setup.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@sherpa-onnx/csrc/qnn/qnn-model.cc`:
- Around line 121-122: The context binary is being duplicated unnecessarily in
qnn-model.cc; use the existing ReadFile result directly instead of copying into
a std::vector<uint8_t>. Update the code around the binary_context_file read path
in qnn-model.cc so the existing char buffer is passed through to the later
buffer.data()/buffer.size() consumers, and keep the handling consistent in the
code paths that use the buffer for QNN setup.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 3ee3fb1c-6b1e-45e0-9e04-0d3973b051dc
📒 Files selected for processing (9)
sherpa-onnx/csrc/CMakeLists.txtsherpa-onnx/csrc/offline-nemo-enc-dec-ctc-model.ccsherpa-onnx/csrc/offline-recognizer-impl.ccsherpa-onnx/csrc/qnn/offline-parakeet-tdt-model-qnn.ccsherpa-onnx/csrc/qnn/offline-parakeet-tdt-model-qnn.hsherpa-onnx/csrc/qnn/offline-recognizer-parakeet-tdt-qnn-impl.hsherpa-onnx/csrc/qnn/qnn-model.ccsherpa-onnx/csrc/qnn/utils.ccsherpa-onnx/csrc/qnn/utils.h
1 participant
See also #3719
Usage
The following shows how to run parakeet-tdt-0.6b-v2 and parakeet-tdt-0.6b-v3 on my Xiaomi 17 Pro with Qualcomm NPU using sherpa-onnx
https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2
Please download a QNN model for it from https://github.com/k2-fsa/sherpa-onnx/releases/tag/asr-models-qnn-binary-2
The following is an example.
Now copy them to your device with Qualcomm NPU and follow
https://k2-fsa.github.io/sherpa/onnx/qnn/run-executables-on-your-phone-binary.html
The following is an example:
Output logs:
https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3
Please download a QNN model for it from https://github.com/k2-fsa/sherpa-onnx/releases/tag/asr-models-qnn-binary-2
The following is an example.
Now copy them to your device with Qualcomm NPU and follow
https://k2-fsa.github.io/sherpa/onnx/qnn/run-executables-on-your-phone-binary.html
The following is an example:
Output logs:
Summary by CodeRabbit