Feature: skip_best_epochs parameter for transfer learning best-model selection

[thhart]

Search before asking

• I have searched the RF-DETR issues and found no similar feature requests.

Description

When using pretrain_weights for transfer learning, epoch 0 evaluates the pretrained model
before any training occurs. BestMetricHolder and EarlyStoppingCallback both treat this
evaluation as a valid candidate for "best model".

If the target dataset is harder or smaller than the pretraining dataset, the pretrained mAP
at epoch 0 may be higher than any subsequent trained epoch — causing the saved
checkpoint_best_total.pth to always be the untrained pretrained weights, and early stopping
to trigger prematurely.

Proposed solution: Add a skip_best_epochs (or warmup_best_epochs) parameter that
excludes the first N epochs from best-model tracking and early stopping evaluation.
For example, skip_best_epochs=3 would:

• Prevent BestMetricSingle.update() from recording a new best during epochs 0–2
• Prevent EarlyStoppingCallback from counting patience during epochs 0–2
• Reset baselines at epoch skip_best_epochs so the first eligible epoch becomes the
  initial reference point

Note: the existing warmup_epochs parameter only affects the LR schedule (linear warmup
before cosine annealing) and does not influence best-model selection or early stopping.

Use case

Industrial transfer learning: we fine-tune RF-DETR (pretrained on COCO) on a domain-specific
dataset with fewer classes and different image characteristics. The pretrained model achieves
~0.84 mAP on epoch 0 (before any training), but training on the new domain initially drops
mAP before recovering. Without skipping early epochs, the "best" checkpoint is always the
untrained pretrained model, and early stopping halts training before the model can adapt.

This affects anyone using pretrain_weights for transfer learning on datasets where the
pretrained model's initial evaluation score is artificially high relative to the training
trajectory. A skip_best_epochs parameter would let users control when best-model tracking
begins, ensuring the saved checkpoint reflects actual training progress.

Additional

Proposed solution:

Add a skip_best_epochs (or warmup_best_epochs) integer parameter to RFDETRConfig
(default 0 for backward compatibility). During training:

1. BestMetricSingle.update(new_res, epoch) returns False for epoch < skip_best_epochs
   — no checkpoint is saved as "best" during warmup
2. EarlyStoppingCallback.update(epoch) skips patience counting for epoch < skip_best_epochs
3. At epoch == skip_best_epochs, best_res is reset to init_res (0.0 or -1.0) so the
   first eligible epoch establishes a fair baseline from actual training, not from pretrained
   evaluation

This would be a small change in BestMetricSingle.update() (util/utils.py) and
EarlyStoppingCallback.update() (util/early_stopping.py), plus the config parameter.

Note: The existing warmup_epochs parameter only affects the LR schedule (linear warmup
before cosine annealing) and does not influence best-model selection or early stopping.
freeze_encoder also does not address this — the issue is in metric tracking, not in
weight updates.

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[isaacrob]

Are you saying that the model is best before it trains at all, and only recovers much later? I'm not sure what you mean that it evaluates at epoch 0 using only pretrained weights. My understanding is that it evaluates after the first training epoch. Using only pretrained COCO weights the mAP should be 0 for any non COCO class. Are you resuming from an existing checkpoint?

Why not just disable early stopping and take the best checkpoint after some sufficiently large number of epochs?

[thhart]

Thanks for the quick response.

First, to clarify what transfer learning means in this context: instead of training a model from scratch (random weights), you take a model that has already learned to detect objects — in this case RF-DETR pretrained on COCO with 80 classes. This pretrained model already understands edges, textures, shapes, spatial relationships, and object boundaries. Transfer learning then fine-tunes this existing knowledge on a new, typically smaller domain-specific dataset. The advantage is massive: you need far less training data and time because the model isn't starting from zero — it's adapting what it already knows.

pretrain_weights loads the full model including the detection head — not just the backbone. When the target dataset contains classes that overlap with COCO (which is common in industrial fine-tuning — e.g. we detect specific types of objects that share visual features with COCO categories), the pretrained model already performs well on those classes before any training.

This is expected behavior: the whole point of transfer learning is that you start from a model that already knows something about your target domain. The problem is what happens during fine-tuning:

1. Epoch 0: evaluates the pretrained model → high mAP (0.84 in our case). BestMetricHolder records this as the best score, checkpoint_best_total.pth is saved.
2. Epochs 1–N: training begins, learning rate disrupts the already-good weights → mAP temporarily drops (e.g. to 0.65). This is the well-known catastrophic forgetting during early adaptation — the model is unlearning COCO generalization while not yet having learned domain-specific features.
3. Epochs N+1 onward: the model recovers and starts learning domain-specific patterns. mAP climbs back: 0.72 → 0.78 → 0.83 → eventually 0.88, surpassing the pretrained baseline.

Here's the core problem: none of the trained models from step 3 are ever saved as best, because BestMetricHolder compares every epoch against the 0.84 from epoch 0. The model at epoch 40 with 0.83 mAP — which is genuinely better for the target domain because it has learned domain-specific features the COCO model never had — is discarded in favor of the untrained COCO weights. The 0.84 at epoch 0 is misleading: it reflects COCO generalization evaluated on a partially overlapping dataset, not actual fitness for the target task. Meanwhile, the 0.83 at epoch 40 reflects true domain adaptation.

It gets worse with early stopping: patience counting starts from the 0.84 baseline at epoch 0, so the "dip" phase exhausts patience and terminates training before recovery even begins. The result: checkpoint_best_total.pth contains untrained pretrained weights, and the model never had a chance to adapt.

Disabling early stopping and running for many epochs works as a workaround, but a skip_best_epochs parameter would solve this properly: ignore the first N epochs for best-model tracking and early stopping, let the model pass through the adaptation dip, then start tracking from a fair baseline that reflects actual training progress. This is a common pattern in fine-tuning frameworks (e.g. PyTorch Lightning's EarlyStopping supports similar warmup concepts).

[isaacrob]

this is not correct. without any finetuning, the model will exclusively detect COCO classes. this will only give non-zero mAP for classes that are literally found in COCO. the benefit of overlapping features is only useful if there's a relevant class embedding vector for your class, which requires finetuning if the class is not one of the COCO classes.

how are you defining a 'better' model if not by any metric? if the model after further finetuning gets a worse metric, it is less useful for detecting objects belonging to that category.

can you post some code to reproduce the issue you're seeing? or maybe some training logs?

I'm not opposed to adding this feature but I think there's a core understanding issue here that adding a new feature doesn't address. though I'm still not sure why you wouldn't just let it run for a certain number of epochs and take the best checkpoint, instead of excluding a certain number of epochs.

as a side note, are you using AI to generate this issue? the tone sounds like an LLM

[thhart]

Hi, I think we need to clarify — COCO here refers to the data format, not the COCO class set. Also, looking back, I mistakenly framed this as transfer learning when it's actually fine-tuning. Our class base is fixed and matches the prior model. We're not changing or extending classes.

What we do is iterative fine-tuning: take the best prior model, continue training with more complex and diverse examples to make the model more generic. The prior checkpoint scores high because it was evaluated on an easier distribution. The harder training examples naturally produce lower mAP initially, even though the resulting model handles more edge cases. Early stopping kills training before the model adapts to the harder distribution, and checkpoint_best_total.pth always points back to the prior weights.

skip_best_epochs would let us get past that initial dip. Alternatively, would you be open to a reset_best_on_resume flag that resets the best metric baseline when loading from pretrain_weights?

PS: Yes, I use an LLM to be more precise in the wording.

[isaacrob]

got it, that makes much more sense. so the issue is that the checkpoint has a stored mAP which was evaluated on a different dataset but with the same classes and you're using resume which loads that checkpoint's best score, which is not comparable because of the domain shift with the new evaluation set? the issue isn't that there's a 'dip' then, it's that the metric isn't comparable.

having said that I still think I am missing something. the 'pretrain_weights' and the 'resume' mechanism are two different features. which are you using? neither of them appear to be loading a prior best_map to carry over into the training period (though I would expect it to in the resume case, which would match the behavior you're describing but I'm not seeing a mechanism to do that when I review the relevant code

rf-detr/src/rfdetr/main.py

Line 409 in c9dede7

best_map_5095 = 0

)