[Bounty $1500] Higgs Audio v2 bring up using TTNN APIs

[tvardhineniTT]

📝 Background

This bounty is for bringing up the Higgs Audio v2 model using TTNN APIs on Tenstorrent hardware (Wormhole or Blackhole).
Higgs Audio v2 is a cutting-edge text-audio foundation model from Boson AI that redefines expressiveness in audio generation. Released in 2025, it offers state-of-the-art capabilities:
Exceptional expressiveness: Industry-leading performance on emotion and prosody benchmarks
Multi-speaker dialog: Natural multi-speaker conversations with distinct voices
Voice cloning: Zero-shot voice cloning from reference audio
Diverse audio generation: Speech, sound effects, music, and environmental sounds
Massive training scale: Trained on 10 million hours of audio data (AudioVerse dataset)
Custom tokenizer: Unified semantic and acoustic tokenization
DualFFN architecture: Enhanced LLM for acoustic modeling with minimal overhead
Apache 2.0 license: Fully open source and commercially usable
The model achieves 75.71% win-rate on EmergentTTS-Eval emotions and best-in-class multi-speaker generation quality.
The goal is to enable this model to run on TT hardware for high-throughput, low-latency audio generation across diverse use cases including virtual assistants, audiobooks, content creation, and interactive media.

🎯 What Success Looks Like

A successful submission will fulfill all requirements in the following stages. Payout is made after all three stages are completed.

Stage 1 — Bring-Up

• Implement Higgs Audio v2 using TTNN APIs (Python)
• Implements the full generation pipeline:
  LLM backbone with DualFFN architecture
  Audio tokenizer (semantic + acoustic features)
  Audio decoder (token-to-waveform conversion)
• Model runs on either N150 or N300 tenstorrent hardware with no errors
• Supports multiple generation modes:
  Text-to-speech: Generate speech from text only
  Voice cloning: Generate speech with reference audio
  Multi-speaker dialog: Generate conversations with distinct speakers
• Produces valid audio output on sample texts (English and multilingual)
• Output is verifiable (audio quality assessment, compare with PyTorch reference)
• Achieves baseline throughput target:
  At least 30 tokens/second for autoregressive generation
  Real-time factor (RTF) < 0.5 for typical sentences
• Accuracy evaluation: Token-level accuracy > 95% against PyTorch reference
• Audio quality: Passes intelligibility and expressiveness tests
• Clear instructions for setup and running the model

Stage 2 — Basic Optimizations

• Use optimal sharded/interleaved memory configs for LLM layers
• Implement efficient sharding strategy for:
  • Token embeddings (text + audio tokens)
  • DualFFN transformer layers
• Multi-head attention mechanisms
• Audio tokenizer encoder/decoder
• Fuse simple ops where possible (e.g., layer normalization, attention patterns, activation functions)
• Store intermediate activations in L1 where beneficial
• Use recommended TTNN/tt-metal LLM flows
• Leverage TT library of fused ops for attention and MLP blocks
• Optimize DualFFN architecture (dual feed-forward networks)
• Efficient KV-cache management for autoregressive generation
• Optimize audio tokenizer integration

Stage 3 — Deeper Optimization

• Maximize core counts used per inference
• Implement deeper TT-specific optimizations:
  - Efficient KV-cache management for long sequences
  - Optimized DualFFN computation (parallel FFN paths)
  - Flash Attention or equivalent for attention layers
• Minimize token generation latency
• Batch processing for multiple utterances/speakers
• Efficient sampling strategies (temperature, top-p, top-k)
• Pipeline audio encoding/decoding with LLM generation
• Minimize memory and TM (tensor manipulation) overheads
• Explore speculative decoding or other acceleration techniques
• Document any advanced tuning, known limitations, or trade-offs
• Target stretched goals:
  - 60+ tokens/second generation speed
  - RTF < 0.2 for real-time applications
• Support for longer contexts (multi-turn dialog)
• Efficient multi-speaker handling

🧭 Guidance & Starting Points

• Use the TTNN model bring-up tech report as your primary reference
• Reference LLM implementations in tt-metal for LLM-based model patterns
• Use the official Higgs Audio v2 repository for model architecture details
• Refer to the Higgs Audio v2 blog for technical details
• Refer to tokenizer blog for audio tokenizer details
• Refer to DualFFN architecture blog for architecture innovations
• Refer to TT Fused ops PR #29236 for optimization opportunities
• The model architecture consists of:
  • Audio tokenizer: Unified semantic and acoustic feature extraction
  • LLM backbone: Transformer with DualFFN architecture
  • DualFFN layers: Dual feed-forward networks for text and audio tokens
  • Audio decoder: Token-to-waveform conversion (24 kHz output)
• Key challenges:
  • DualFFN architecture optimization (dual FFN paths)
  • Large vocabulary (text + audio tokens)
  • Autoregressive generation latency
  • Audio tokenizer integration
  • Multi-speaker context management
• Ask for help or file issues if ops are missing in TTNN

🔎 Possible Approaches

• Start from the official Higgs Audio v2 repository and port components sequentially:
• End-to-end pipeline integration
• Validate each submodule's output against PyTorch reference before integration
• For example the LLM backbone:
• Start with standard transformer layers
• Add DualFFN architecture (key innovation)
• Optimize attention mechanisms
• Implement efficient KV-cache
• Experiment with different sharding strategies
• Use TTNN profiling tools to identify bottlenecks
• Test diverse use cases
• Open a draft PR early to get feedback on your approach

📊 Result Submission Guidelines

Beyond the model implementation itself, contributors must submit the following material as proof of work.
However, feel free to open a PR at any time if you want us checking that you are on the right track.
Just understand that payout is only made after all 3 stages are completed.

Deliverables:

• Functional model implementation
• Validation logs (output correctness)
• Performance report + header for final review

Links:

• Performance Sheet
• Perf Header Docs

📚 Resources

Model Resources

• Higgs Audio v2 Official Repository
• Higgs Audio v2 Release Blog
• Audio Tokenizer Technical Blog
• DualFFN Architecture Blog
• Boson AI Website

Evaluation Resources

• Seed-TTS Eval Benchmark
• Emotional Speech Dataset (ESD)
• EmergentTTS-Eval Paper

TT-Metal Resources

• TTNN Model Bring-up Tech Report
• LLM Implementations in tt-metal
• TT Fused Ops PR #29236
• Performance Report Header

[IshGadol]

Hi @tvardhineniTT and Tenstorrent team —

I'd like to be assigned to this bounty.

Background
I'm already active in the TTNN ecosystem (see my work and comments on the Faster-RCNN bring-up and related profiling discussions).
My local environment is fully operational on TT hardware via Koyeb (N300 instance) with verified TTNN / tt-metal support.

Intent
I plan to begin with a Stage 1 bring-up focused on correctness and reproducibility:

• Port the Higgs Audio v2 architecture to TTNN using the DualFFN and tokenizer modules from the Boson AI repository.
• Validate each sub-module against its PyTorch reference outputs.
• Produce logs, validation traces, and setup documentation aligned with Tenstorrent’s model bring-up report template.

If, after completing my exploratory work, I find that key operators are missing or that the integration path isn’t yet practical within TTNN, I’ll document the blockers clearly and step aside so others can continue without disruption.

Goal
Deliver a working end-to-end pipeline that meets Stage 1 success criteria (functional generation, verifiable audio output, no CPU fallbacks) and establish a solid foundation for later optimization work.

Please confirm assignment when convenient.
Thanks for continuing to make these bounties accessible — I look forward to contributing again.

— David Schutt (@IshGadol)

[jberkowitzTT]

Thanks @IshGadol for the interest, and I always look forward to seeing your contributions! Thank you for all of the hard work you've done already, and I'm eager to see what you do for this one! I see that you're working on a couple of different bounties as the moment, so feel free to let me know at any point the 3 become to much to handle at once.

The bounty is now yours! Please update this issue with your progress within 2 weeks.

Review our Bounty Terms & Conditions, as these terms apply. Good luck with the project, and let us know if there are any bugs/issues on our side that are stopping you from completing the bounty.

Please note that Tenstorrent is a US based company. As such, if you are an non-US resided participant and work is performed in the US, a 30% withholding may be deducted upon payout for tax withholding, unless your country of residence has a tax treaty with the US.

[IshGadol]

Hi @jberkowitzTT — thank you for the assignment and for the continued trust. I appreciate it.

I confirm I’m taking on the Higgs Audio v2 bounty and will provide a concrete progress update within the 2-week window.

I’m being mindful of scope across the bounties I’m currently working, and I’ll approach this one in a staged, disciplined way:
• Stage 1 correctness-first bring-up (end-to-end pipeline, no CPU fallbacks),
• strict parity validation against PyTorch reference,
• and early visibility via repo + updates so alignment stays tight.

I’ll surface any missing ops, TTNN gaps, or hardware-related blockers immediately so expectations remain clear.

Looking forward to digging into this — thanks again.

[jberkowitzTT]

Hi @IshGadol ! I hope the new has started off great, and I wanted to check in to see if you happened to have any updates on this issue or the other bounties you have assigned

[Bounty $1500] Add model: Stable Diffusion 1.4 (512x512)

[Bounty $1500] Faster-RCNN bring up using TTNN APIs

[minaliuTT]

Hi @IshGadol, Just checking in on your progress with this bounty. Any updates or blockers to share since your last report Thanks!