Number of training iterators is no longer impacted by 'grad_accum_steps'

[LeMinhNgan]

Search before asking

• I have searched the RF-DETR issues and found no similar bug report.

Bug

There is an abnormal thing that I observed after we switched from 1.5.x and older version to PTL trainer is the Number of training iterators show in progress bar during training steps is independent from grad_accum_steps . For example: if i have a train folder contains 8000 images, batch_size= 2 and grad_accum_steps= 4

• in v1.5.x and before: n_iterator = 8000/(batch_size*grad_accum_steps) = 1000
• From v1.6.x: n_iterator = 8000/(batch_size) = 4000. The grad_accum_steps no longer participate
  I used to use this parameter to adjust the work load of GPU VRAM during training . I wonder if we forgot something when switching to PTL

Environment

• OS: windows 11
• RTX 5090
• cuda 12.8
• rfdetr: 1.6.x
• python 3.10

Minimal Reproducible Example

model= RFDETRLarge(resolution= cfg.resolution, device= cfg.model_device, num_classes= num_classes)       
model.train(dataset_dir= cfg.dataset_dir, epochs= 100, batch_size= 2, grad_accum_steps= 4)

Additional

No response

Are you willing to submit a PR?

• Yes, I'd like to help by submitting a PR!

[Borda]

Hey there! Thanks for bringing this up and detailing your observation.

You have a sharp eye—this is a very common point of confusion when migrating custom training loops over to PyTorch Lightning (PTL). The short answer is: nothing is broken, and your VRAM is still being protected. PyTorch Lightning simply tracks progress differently than our old custom loop did.

Here is exactly what is happening under the hood:

1. Why the Progress Bar Changed

• In v1.5.x and older (Pre-PTL): Our old progress bar was programmed to track optimizer steps (or "effective batches"). With 8000 images, a physical batch size of 2, and accumulating over 4 steps, your optimizer only updated 1000 times (8000 / 2 / 4 = 1000).
• In v1.6.x (PTL): PyTorch Lightning’s default progress bar tracks forward passes (i.e., dataloader batches), not optimizer steps. Since your batch_size is 2, the dataloader yields 4000 physical batches (8000 / 2 = 4000). The progress bar ticks forward for every single batch that goes through the model.

2. Is GPU VRAM Workload Affected?

No, your VRAM is perfectly safe. Peak GPU VRAM usage is dictated entirely by the physical batch_size loaded into the model at one time (which is still 2).

grad_accum_steps doesn't dictate VRAM; it only acts as a delay multiplier to create an effective batch size for the optimizer (2 * 4 = 8) to stabilize model convergence. Because your physical batch size is still 2, your GPU will process the exact same lightweight workload per forward pass as it did in v1.5.x.

3. Making sure nothing was forgotten in the switch

Even though the progress bar shows 4000 iterations, PyTorch Lightning is still handling the gradient accumulation internally. It processes a batch of 2 (forward + backward pass), accumulates the gradients, updates the progress bar, and repeats. Only on every 4th iteration does it actually perform optimizer.step() and optimizer.zero_grad().

To ensure the math is working as intended, we just need to verify that our config's accumulation parameter is being passed to PyTorch Lightning's accumulate_grad_batches argument. In the trainer initialization, it looks like this:

from lightning.pytorch import Trainer

trainer = Trainer(
    # This is the PTL argument that handles your grad_accum_steps
    accumulate_grad_batches=config.grad_accum_steps, 
    # ... other arguments like max_epochs, devices, etc.
)

Tip: If you want proof during runtime that it's working, you can log PyTorch Lightning's trainer.global_step property. global_step tracks actual optimizer steps—you will see it correctly reach 1000 at the end of the epoch, even while the main progress bar hits 4000.

I hope this clears up the discrepancy! Let me know if you have any other questions about the PTL migration. 🦝

[LeMinhNgan]

Thank you for detail explanation.