[Stage 2] Difficulty Reproducing Text-to-Motion FID Scores on A6000 (12.81 vs 11.79) - Hardware/Precision Inquiry

[bring-nirachornkul]

Dear Authors,

Thank you for your impressive work on MotionStreamer. We are currently working on reproducing the results for the Stage 2 (Text-to-Motion) model using your official codebase and instructions, but we are observing a consistent gap in the FID score compared to the paper.

1. Experimental Setup We adhered strictly to the provided training scripts and configurations:

Codebase: Original train_t2m.py (Stage 2).

Stage 1 Checkpoint: We used the official pre-trained casual_TAE checkpoints provided in the repository.

Hardware: 8x NVIDIA A6000 GPUs.

Hyperparameters: Batch size 32 per GPU (Global Batch Size = 256), Learning Rate 1e-4, 100k Iterations.

2. Results Comparison While the downloaded checkpoints match the paper perfectly, our reproduction from scratch consistently lags behind by ~1.0 FID.

Model Source	FID ↓	R@1 ↑	MM-D (Real) ↓
Paper Reported	11.79	0.631	15.15
Downloaded Checkpoint	11.80	0.631	15.15
Our Reproduction (A6000)	12.81	0.635	15.15

3. Questions Since we are using the exact same code and global batch size, we suspect this might be related to hardware precision differences (e.g., A6000 defaults to TF32 whereas older GPUs might use true FP32).

Hardware: Could you please share which specific GPUs (and how many) were used for the final experiments reported in the paper?

Precision: Did you explicitly disable TF32 (allow_tf32 = False) or use specific mixed-precision settings that might not be default in the current script?

Any insights on closing this gap would be greatly appreciated.

Thank you!

Bring Nirachornkul

[InvertedForest]

Hi,

I'm also reproducing this codebase. Could you please report the loss at the last iteration?

Thanks.

[bring-nirachornkul]

H200 (2 GPUS x 128 = 256 for the batch size) @239,900 iterations

2026-01-26 15:23:30,443 INFO Train. Iter 239900: Loss 0.00465 | Data: 0.5ms, Text: 1048.9ms, Fwd: 375.0ms, Bwd: 412.7ms | VRAM Alloc: 22.73GB, Res: 111.74GB
2026-01-26 15:23:30,443 INFO Range 239801-239900: Avg 1.7684 sec/iter

Loss : 0.4 %
Still training

Here is the graph :

H200 (2 GPUS x 128 = 256 for the batch size) @500,000 iterations, BF16

2026-01-24 07:12:29,027 INFO Train. Iter 500000: Loss 0.00191 | Data: 2.6ms, Text: 211.4ms, Fwd: 81.7ms, Bwd: 119.1ms | VRAM Alloc: 22.73GB, Res: 121.94GB
2026-01-24 07:12:29,028 INFO Range 499901-500000: Avg 0.4551 sec/iter

Loss : 0.4 %
FID: 22.112861525936296 → 22.1129
Diversity (Real): 27.0572
Diversity (Pred): 27.098692 → 27.0987
R@k: from Pred (top1/top2/top3)
MM-D (Real): 15.150793075561523 → 15.1508
MM-D (Pred): 19.27187728881836 → 19.2719

Here is the graph :

RTX A6000 (8 GPU x 32 = 256) @200,000 iterations

2026-01-25 02:46:18,887 INFO Train. Iter 200000: Loss 0.00837 | Data: 1.7ms, Text: 953.3ms, Fwd: 261.2ms, Bwd: 251.7ms | VRAM Alloc: 22.67GB, Res: 46.20GB
2026-01-25 02:46:18,888 INFO Range 199901-200000: Avg 1.4430 sec/iter

Loss : 0.8 %
FID: 16.39755926288649 → 16.3976
Diversity (Real): 27.2995
Diversity (Pred): 27.343843 → 27.3438
R@k: taken from Pred (top1/top2/top3)
MM-D (Real): 15.150793075561523 → 15.1508
MM-D (Pred): 16.518611907958984 → 16.5186

Here is the graph :

[InvertedForest]

I just finished the reproduction. The training code strictly followed the readme, with batchsize 8x32 on 8 4090 using "accelerate launch --num_processes $NUM_GPUS --mixed_precision bf16 XXX". I also cached feat_text to save gpu memory. The results are as follows:

# official
loading transformer checkpoint from Experiments/t2m_model/latest.pth 
2026-01-26 20:29:21,616 INFO fid: [11.753709920368237]
2026-01-26 20:29:21,616 INFO div: [27.327421188354492]
2026-01-26 20:29:21,617 INFO top1: [0.641796886920929]
2026-01-26 20:29:21,617 INFO top2: [0.7972656488418579]
2026-01-26 20:29:21,617 INFO top3: [0.8617187738418579]
2026-01-26 20:29:21,617 INFO MM-dist (matching score) : [15.936205863952637]

# reproduce
loading transformer checkpoint from Experiments/t2m_model_init/100000.pth  
2026-01-27 11:02:16,255 INFO final result:  
2026-01-27 11:02:16,255 INFO fid: [11.872806171763727]  
2026-01-27 11:02:16,255 INFO div: [26.735172271728516]  
2026-01-27 11:02:16,255 INFO top1: [0.6343749761581421]  
2026-01-27 11:02:16,255 INFO top2: [0.797656238079071]  
2026-01-27 11:02:16,255 INFO top3: [0.8695312738418579]  
2026-01-27 11:02:16,255 INFO MM-dist (matching score): [15.902449035644532] 

The FID is already very close to the paper’s reported value. I noticed you were trying to train longer to get lower diffusion loss, but FID and diffusion loss are not strongly correlated—this can actually lead to overfitting.

Moreover, the diffusion loss of mine is 0.015, while the official diffusion loss I reproduced with its ckpt is 0.0097. This is indeed a bit strange. I hope there’s no bug here.

[bring-nirachornkul]

I just finished the reproduction. The training code strictly followed the readme, with batchsize 8x32 on 8 4090 using "accelerate launch --num_processes $NUM_GPUS --mixed_precision bf16 XXX". I also cached feat_text to save gpu memory. The results are as follows:

``` # official loading transformer checkpoint from Experiments/t2m_model/latest.pth 2026-01-26 20:29:21,616 INFO fid: [11.753709920368237] 2026-01-26 20:29:21,616 INFO div: [27.327421188354492] 2026-01-26 20:29:21,617 INFO top1: [0.641796886920929] 2026-01-26 20:29:21,617 INFO top2: [0.7972656488418579] 2026-01-26 20:29:21,617 INFO top3: [0.8617187738418579] 2026-01-26 20:29:21,617 INFO MM-dist (matching score) : [15.936205863952637]

reproduce

loading transformer checkpoint from Experiments/t2m_model_init/100000.pth
2026-01-27 11:02:16,255 INFO final result:
2026-01-27 11:02:16,255 INFO fid: [11.872806171763727]
2026-01-27 11:02:16,255 INFO div: [26.735172271728516]
2026-01-27 11:02:16,255 INFO top1: [0.6343749761581421]
2026-01-27 11:02:16,255 INFO top2: [0.797656238079071]
2026-01-27 11:02:16,255 INFO top3: [0.8695312738418579]
2026-01-27 11:02:16,255 INFO MM-dist (matching score): [15.902449035644532]

The FID is already very close to the paper’s reported value. I noticed you were trying to train longer to get lower diffusion loss, but FID and diffusion loss are not strongly correlated—this can actually lead to overfitting.

Moreover, the diffusion loss of mine is 0.015, while the official diffusion loss I reproduced with its ckpt is 0.0097. This is indeed a bit strange. I hope there’s no bug here.

Hi Tao,

Thanks for the insight! Your theory on overfitting is spot on. My data confirms it:

A6000 (100k iters): FID 12.47 (Loss ~0.012) — Success

A6000 (200k iters): FID 16.40 (Loss ~0.008) — Overfitted

H200 (500k iters): FID 22.11 (Loss ~0.002) — Broken

To fix my H200 run and match your results (FID ~11.8), I have 4 quick questions:

Per-GPU Batch Sensitivity: I matched your Global Batch (256) using 2x H200s (128 batch/GPU) vs your 8x 4090s (32 batch/GPU). Do you suspect this model is sensitive to per-device batch statistics (needs "noisier" small batches)?

The "Magic" Loss: You mentioned the successful run had a loss of ~0.015. Do you treat this value as a hard stopping target, or was 100k just a standard stop that happened to align there?

Original Checkpoint Data: You noted the official checkpoint has a loss of 0.0097. Did you derive this by running the official weights through a validation script yourself?

Offline Text: Just to confirm, by "offline text," you mean pre-computing T5 embeddings to disk (like the standard get_latent.py workflow) to save VRAM?

Thanks again—this saved me from training for 500k iterations!

Sincerely,

Bring

[InvertedForest]

• Per-GPU Batch Sensitivity: Theoretically, with DDP and mean loss, there is no impact. However, due to the BF16 precision, batch normalization, dropout, etc., I'm not quite sure. You can try it with 8 gpus and bf16.
• The "Magic" Loss: 100k just a standard stop
• Original Checkpoint Data: resume training with the official ckpt and lr=0
• Offline Text: Yes

Based on my experience, it is normal for FID to fluctuate. Even for different training seeds with the same configuration, the optimal cfg strength (default: 4) can vary. In addition, it is recommended that you save the ckpt during the training process. You can compare the differences between 90,000 and 100,000 epoch.

[bring-nirachornkul]

Hi Tao,

Thanks for the tip on using lr=0 to check the loss—that saved us a lot of time.

I have an update that strongly supports your suspicion about Per-GPU Batch Sensitivity. We ran a controlled experiment on H200s keeping the Global Batch Size constant at 256 but varying the Per-GPU Batch Size:

• 2x H200 (Batch 128/gpu): FID stalled at ~22.0 (Iter 60k).
• 4x H200 (Batch 64/gpu): FID improved to ~15.6 (Iter 60k).
• 8x A6000 (Batch 32/gpu): FID reached ~12.5 (My previous run).

H200 Benchmarks (All runs using Global Batch 256)

GPU Setup	Precision	Text Mode	Batch Size (Global / Per GPU)	Iterations	Loss	FID ↓	Result Analysis
H200 (2x)	BF16	Online	256 / 128	500,000	0.19	22.11	Broken (Overfitted)
H200 (2x)	FP32	Online	256 / 128	30,000	1.4	27.95	Early Training
H200 (2x)	FP32	Online	256 / 128	40,000	1.3	22.52	Improving
H200 (2x)	FP32	Online	256 / 128	50,000	1.2	23.91	Unstable / Worsening
H200 (2x)	FP32	Online	256 / 128	60,000	1.0	22.03	Stalled (High Batch Size limit)
H200 (4x)	FP32	Offline	256 / 64	50,000	1.7	18.59	Better than 2x Setup
H200 (4x)	FP32	Offline	256 / 64	60,000	1.5	15.61	Significant Improvement

It seems this architecture degrades significantly when the per-device batch exceeds 64. I am now switching my H200 run to use 8 GPUs (or gradient accumulation) to mimic the 32-batch "sweet spot."

To finalize my setup, I have a few specific questions:

1. Gradient Accumulation vs. Physical Batches
Since per-device batch size clearly matters (128 vs 32), do you think Gradient Accumulation (e.g., Batch 32 accumulated 8 times) is mathematically equivalent for this specific model? Or does the model actually need the statistical noise of small physical batches (e.g. for BatchNorm statistics) to generalize well?

2. Overfitting Mechanics
It is fascinating (and frustrating) that lower loss often leads to worse FID here. In your experience with this codebase, is this overfitting coming from the VAE reconstruction part (memorizing poses) or the Diffusion part (memorizing noise)? Which component do you think breaks first?

3. Code & Config Details

• Code Mods: Did you modify any underlying code to achieve your reproduction, or is it strictly the original repo with train_t2m.py?
• CFG Automation: Do you manually search for the optimal CFG scale (e.g., checking 2.0, 3.0, 4.0) for every checkpoint, or do you have a script that sweeps this automatically?
• Loss Scale: In my logs, the loss values are scaling differently (e.g., 1.5 vs 0.015). Did you apply any specific loss scaling factor in your logging, or is this just a difference in reduction='mean' vs sum?
• Hyperparameters: Did you change any other hyperparameters (LR, seed, etc.)? I am using the default seed: 123 and lr: 0.0001.

For reference, here is the config I am currently running on my 4x H200 setup (Batch 64/GPU):

{
    "batch_size": 64,
    "lr": 0.0001,
    "num_gpus": 4,
    "optimizer": "adamw",
    "seed": 123,
    "total_iter": 100000,
    "weight_decay": 1e-06
}

Thanks again, your insights have been incredibly helpful.

Best,
Bring

[InvertedForest]

1. Gradient Accumulation vs. Physical Batches: Theoretically, under the configuration of (DDP + mean loss + fixed overall batch size), changing the number of GPUs should not lead to differences. I can only attribute these varying performances to precision issues, especially since I am using bf16, which indeed introduce noise and may affect generalization.
2. Overfitting Mechanics. VAE part is frozen, diffusion with FP32 may be easier to overfit. According to your table, FP32 may have an optimal stopping point earlier than 100,000, such as 80,000.
3. Code. I didn't change anything except for the bf16 and cache feat text of accelerate.

2026-01-26 22:56:53,025 INFO {
    "batch_size": 32,
    "dataname": "t2m_272",
    "decay_option": "all",
    "depth": 3,
    "dilation_growth_rate": 3,
    "down_t": 2,
    "exp_name": "t2m_model_init",
    "gamma": 0.05,
    "hidden_size": 1024,
    "latent_dim": 16,
    "latent_dir": "humanml3d_272/t2m_latents",
    "lr": 0.0001,
    "mode": "pos",
    "num_diffusion_head_layers": 9,
    "num_gpus": 8,
    "optimizer": "adamw",
    "out_dir": "Experiments/t2m_model_init",
    "resume_pth": null,
    "resume_trans": null,
    "seed": 123,
    "stride_t": 2,
    "text": "A man is jogging around.",
    "total_iter": 100000,
    "weight_decay": 1e-06
}

I think FID = 12 is already quite good. There are indeed too many variables, and I can't guarantee that I can perfectly reproduce it under different seeds : )

[bring-nirachornkul]

1. Gradient Accumulation vs. Physical Batches: Theoretically, under the configuration of (DDP + mean loss + fixed overall batch size), changing the number of GPUs should not lead to differences. I can only attribute these varying performances to precision issues, especially since I am using bf16, which indeed introduce noise and may affect generalization.
2. Overfitting Mechanics. VAE part is frozen, diffusion with FP32 may be easier to overfit. According to your table, FP32 may have an optimal stopping point earlier than 100,000, such as 80,000.
3. Code. I didn't change anything except for the bf16 and cache feat text of accelerate.

2026-01-26 22:56:53,025 INFO {
    "batch_size": 32,
    "dataname": "t2m_272",
    "decay_option": "all",
    "depth": 3,
    "dilation_growth_rate": 3,
    "down_t": 2,
    "exp_name": "t2m_model_init",
    "gamma": 0.05,
    "hidden_size": 1024,
    "latent_dim": 16,
    "latent_dir": "humanml3d_272/t2m_latents",
    "lr": 0.0001,
    "mode": "pos",
    "num_diffusion_head_layers": 9,
    "num_gpus": 8,
    "optimizer": "adamw",
    "out_dir": "Experiments/t2m_model_init",
    "resume_pth": null,
    "resume_trans": null,
    "seed": 123,
    "stride_t": 2,
    "text": "A man is jogging around.",
    "total_iter": 100000,
    "weight_decay": 1e-06
}

I think FID = 12 is already quite good. There are indeed too many variables, and I can't guarantee that I can perfectly reproduce it under different seeds : )

Hi Tao,

This is incredibly helpful. Your hypothesis about BF16 precision acting as a regularizer makes perfect sense and explains why my "cleaner" FP32 runs overfitted so quickly.

I now have a clear path forward. I am proceeding with the following setup to mimic your "noise" conditions:

Hardware: 4x A6000 (BF16 Enabled)

Batch: 64 per GPU (Global 256)

Strategy: Checkpointing every 5k iterations and monitoring the 80k–100k window closely.

I will let you know if this finally cracks the 12.0 FID barrier. Thanks again for guiding me away from the 500k training trap!

Best, Bring