Multiple bug fixes: long video denoising, audio parity, upscaler artifacts

[altdotboy]

Hi! I've been working extensively with LTX-2-MLX on Apple Silicon and fixed several significant bugs. Sharing them here in case they're useful for the project.

Fork with all fixes

https://github.com/altdotboy/LTX-2-MLX

Bug Fixes

1. Denoise mask broadcasting — breaks all videos longer than a few seconds

The denoise mask has shape (B, T) but the latent is (B, T, D). Direct multiplication causes incorrect broadcasting, and the error accumulates across timesteps on longer videos. Short clips (1-2s) mask the issue.

Fix: mx.expand_dims(mask, axis=-1) in noisers.py for both GaussianNoiser and DeterministicNoiser.

2. Audio VAE decoder parity (0.999 correlation with PyTorch)

Five fixes in the audio decoder:

• Causal axis drop in Upsample2d (matches PyTorch CausalUpsample)
• Added PixelNorm (norm_out) before final conv (matches _finalize_output)
• Fixed per_channel_statistics weight path
• Fixed upsample conv weight path
• Vocoder: LeakyReLU slope corrected from LRELU_SLOPE (0.1) to 0.01 before conv_post
• Vocoder: Native MLX dilation instead of manual kernel expansion

3. Spatial upscaler GroupNorm — grid/checkerboard artifacts

GroupNorm was normalizing per-frame (B*T, H, W, C) instead of across all frames jointly for 5D tensors. Added group_norm_5d() that normalizes over temporal+spatial dimensions together. Eliminates grid artifacts.

4. av_ca_timestep_scale_multiplier — speech audio nearly silent

Value was 1 instead of 1000, making audio-video cross-attention gate factor 0.001 instead of 1.0. Effectively zeroed cross-modal gates carrying speech information.

5. FPS mismatch — audio latent shape errors

Default fps was 24.0 but should be 25.0 (matching PyTorch frame_rate). Caused audio latent shape mismatch (34 vs 33 frames).

6. Audio weight loading

Added 'a2v' to include_audio filter so audio weights actually load.

New Features

• CFG guidance rescale (rescale_noise_cfg) to prevent oversaturation
• Audio-to-video two-stage pipeline
• Retake pipeline
• Text+image-to-video HQ pipeline
• Channelwise audio noise normalization for duration-independent amplitude
• Gemma 3 text encoder improvements

Testing

All fixes tested on M5 128GB with Qwen3.5-35B. Audio decoder achieves 0.9994 correlation with PyTorch, vocoder 0.9989.

Happy to submit individual PRs for any of these if that's preferred. 🤙

[altdotboy]

Known Issue: Speech audio degrades after ~5 seconds (ambient audio unaffected)

One issue we haven't resolved yet — speech audio (talking characters) degrades into garbled noise after approximately 5 seconds on MLX. Lip sync breaks down in lockstep.

Key finding: Ambient/environmental audio (ocean waves, birds, wind) plays cleanly for 20+ seconds with no degradation. The issue is specific to speech synthesis.

This points to the transformer struggling with the temporal coherence demands of speech (phoneme sequences, prosody, frame-to-frame continuity) rather than a general audio fidelity problem. PyTorch MPS produces clean 10s+ speech with the same weights.

What we've ruled out:

• Audio VAE decoder shapes (identical to PyTorch)
• Vocoder input/output (identical)
• RoPE positions (both use seconds)
• RMSNorm precision (float32 promotion didn't help)
• RoPE float64 frequency grid (made it worse)
• Noise distribution differences (scaling to match PyTorch bf16 std didn't help)

Likely culprit: Something in mx.fast.scaled_dot_product_attention vs PyTorch's SDPA handling long-sequence audio tokens, or accumulated numerical drift across the 48 transformer blocks that only matters for the precision speech requires.

Full investigation log is in the fork: AUDIO_ISSUES.md

[Acelogic]

Huge thanks @altdotboy — this is an outstanding writeup and an even better set of fixes. 🙏

Ported in 3ce9ab7 (cherry-picked from your fork to preserve attribution) plus a small follow-up in c4f8c44 to make the AV path in scripts/generate.py use fps=25.0 (the inline config there was still hardcoded to 24.0 despite the new comment noting 25.0 is required for audio latent parity).

Specifically landed on main:

• Denoise mask broadcasting fix in noisers.py
• All five audio VAE decoder / vocoder parity fixes
• Spatial upscaler group_norm_5d
• av_ca_timestep_scale_multiplier=1000 at the AV load site
• fps + audio weight loading fixes
• CFG guidance rescale, a2vid_two_stage, retake, ti2vid_hq pipelines
• Channelwise audio noise normalization, Gemma 3 connector/encoder updates
• AUDIO_ISSUES.md so the speech-degradation investigation doesn't get lost

Really appreciate the thoroughness of the diagnosis and the offer to split into individual PRs — bundling it actually made review and porting easier here. The 0.999 audio decoder correlation result is impressive work.

Leaving this issue open since the >5s speech degradation is still unresolved upstream — happy to keep collaborating on it.

[dmlsrc]

On the fps=25.0 question: I traced this back to the official Lightricks source and all three reference implementations (Lightricks/LTX-2, Blaizzy's mlx-video, diffusers pipeline) all use 24.0. The 25.0 appears to have come from VideoPixelShape's default matching PAL broadcast rate rather than the model's native rate. Details and reference links in #18.

Also fixed in #18:

• Audio position fp16 precision (echo artifacts)
• round() -> ceil() for audio latent sizing (last frame drop)
• Both improve lip sync on longer clips
• --duration flag for specifying clip length in seconds