Subject: Inquiry about Flow Matching Export for Alpamayo-R1-10B on TensorRT Edge-LLM

[MINTPIPERAS]

Dear NVIDIA TensorRT Edge-LLM Team,

Thanks for your splendid works !

I am currently working on deploying the Alpamayo-R1-10B Vision-Language-Action (VLA) model on a Jetson Thor (Jetpack 7.0) device using TensorRT Edge-LLM. I have successfully built the engines and run inference, but I have a question regarding the inference mode and potential optimization paths.

According to the official documentation and blog posts, Alpamayo-R1-10B integrates a flow matching trajectory decoder to generate future trajectories. The model's config.json also confirms the use of alpamayo_r1.diffusion.flow_matching.FlowMatching. However, based on my understanding of the current TensorRT Edge-LLM workflow (export → build → inference), the VLM backbone (onnx/llm) appears to be exported and run as a standard causal language model, and the current runtime seems to support autoregressive decoding.

My questions are:

1. Is the Alpamayo-R1-10B inference in TensorRT Edge-LLM currently limited to autoregressive decoding? Or does the current action_inference pipeline internally handle the flow matching decoding for the action expert?
2. If flow matching is not yet supported in the current release, could you please provide guidance on how to export and build the model to leverage the flow matching decoder for inference? Are there specific export flags, builder configurations, or a different pipeline (e.g., using the diffusion-based action decoder separately) that I should follow?

I have included my current performance summary and build commands below for reference, as I am actively working on latency optimization.

Thank you for your time and support. I look forward to your guidance.

Best regards,
Piperas From XMU

---

Appendix: Current Setup and Performance

Platform: Jetson Thor (Jetson 7.0 128GB-GPU)

Performance Summary:

=== Performance Summary ===
=== LLM Prefill ===
Reused Tokens: 0
Computed Tokens: 1582
Average Tokens per Run: 1582.00
Average Time per Run: 282.6949 ms
Tokens/Second: 5596.1
Average Time per Token: 0.1787 ms
LLM Prefill - Total Runs: 1, Total GPU Time: 282.69 ms, Average: 282.69 ms
=== LLM Generation (Excluding sampling after prefill) ===
Generated Tokens: 64
Average Tokens per Run: 64.00
Tokens/Second: 15.3
Average Time per Token: 65.4646 ms
LLM Generation - Total Runs: 63, Total GPU Time: 4189.74 ms, Average: 66.50 ms
=== Multimodal Processing ===
Total Images: 16
Total Image Tokens: 1456
Total Multimodal Tokens: 1456
Average Time per Token: 0.0882 ms
Vision Encoder - Total Runs: 1, Total GPU Time: 128.38 ms, Average: 128.38 ms
=== Memory Usage ===
Peak Unified Memory: 14930.82 MB (15656099840 bytes)
=====================================

Build Commands Used:

# LLM engine
./build/examples/llm/llm_build \
  --onnxDir /workspace/user_ws/tensorrt-edgellm-workspace/Alpamayo-R1-10B/onnx/llm \
  --engineDir /workspace/user_ws/tensorrt-edgellm-workspace/Alpamayo-R1-10B/engines_low_batch/llm \
  --maxInputLen 3424 \
  --maxKVCacheCapacity 4096 \
  --maxBatchSize 1

# Visual encoder engine
./build/examples/multimodal/visual_build \
  --onnxDir /workspace/user_ws/tensorrt-edgellm-workspace/Alpamayo-R1-10B/onnx/visual \
  --engineDir /workspace/user_ws/tensorrt-edgellm-workspace/Alpamayo-R1-10B/engines_low_batch \
  --minImageTokens 80 \
  --maxImageTokens 9216 \
  --maxImageTokensPerImage 96

# Action expert engine
./build/examples/multimodal/action_build \
  --onnxDir /workspace/user_ws/tensorrt-edgellm-workspace/Alpamayo-R1-10B/onnx/action \
  --engineDir /workspace/user_ws/tensorrt-edgellm-workspace/Alpamayo-R1-10B/engines_low_batch \
  --maxBatchSize 1

And About This Page：https://nvidia.github.io/TensorRT-Edge-LLM/developer_guide/software-design/llm-inference-runtime.html
In theory, can I use this method to speed up Alpamayo?