Due diligence
Topic
The Rust implementation
Question
Could the team prioritize adding INT4 (Q4_0/Q4_K) quantization support specifically for the moshi-backend Rust implementation, alongside optimized ARM NEON/SIMD kernels for the Mimi codec and Temporal Transformer layers?
I am developing an offline, full-duplex assistant running on a Raspberry Pi 5 paired with the Hailo-10H NPU (40 TOPS).
To achieve a great user experience, the system must maintain Moshi's native 12.5Hz (80ms per frame) cadence without thermal throttling or audio stutter. While the Hailo-10H features 8GB of dedicated memory, the 34 GB/s LPDDR4 bandwidth creates a hard physical bottleneck for a 7.6B model in INT8 (8GB * 12.5Hz$ far exceeds the bus capacity).
To maintain real-time streaming, we must reduce the weight-fetch payload to a 4-bit profile.
Excited to hear from you.
Due diligence
Topic
The Rust implementation
Question
Could the team prioritize adding INT4 (Q4_0/Q4_K) quantization support specifically for the moshi-backend Rust implementation, alongside optimized ARM NEON/SIMD kernels for the Mimi codec and Temporal Transformer layers?
I am developing an offline, full-duplex assistant running on a Raspberry Pi 5 paired with the Hailo-10H NPU (40 TOPS).
To achieve a great user experience, the system must maintain Moshi's native 12.5Hz (80ms per frame) cadence without thermal throttling or audio stutter. While the Hailo-10H features 8GB of dedicated memory, the 34 GB/s LPDDR4 bandwidth creates a hard physical bottleneck for a 7.6B model in INT8 (8GB * 12.5Hz$ far exceeds the bus capacity).
To maintain real-time streaming, we must reduce the weight-fetch payload to a 4-bit profile.
Excited to hear from you.