Differences in Decoder Fine-Tuning Between Indoor and Outdoor Scenarios

[Garfield2005]

From Tables 3 and 4 in the Utonia paper, it can be observed that:

1. For indoor scenarios: fine-tuning the decoder (dec) with Concerto or Utonia improves the results, outperforming PTv3.
2. For outdoor scenarios: fine-tuning only the decoder (dec) with Concerto or Utonia leads to worse results, underperforming PTv3.

How can this phenomenon be explained?

Is it reasonable to conclude that for outdoor scenarios, if performance is the priority, it is not recommended to fine-tune only the decoder?

[Yujia-Zhang0913]

Hi
In my view, outdoor LiDAR point clouds have noticeable range-dependent sparsity and specific beam patterns. A frozen encoder works fine on uniform data, but for outdoor datasets, an unfrozen encoder really helps the model adapt to those patterns. If performance is the top priority, I'd recommend fine-tuning both the encoder and the decoder.

Moreover, you can also further fine-tune with additional datasets using PPT.

[Garfield2005]

I’m using concerto_large_outdoor.pth, which is already pretrained on outdoor scenes, so hasn’t it already learned outdoor-specific features during pretraining?

[Yujia-Zhang0913]

Yes, it has already learned outdoor-biased features. However, this does not imply that a frozen encoder with decoder-only tuning is sufficient for every outdoor downstream benchmark. A model can be pretrained on outdoor scenes and acquire strong outdoor priors, while still requiring encoder adaptation for a specific downstream segmentation task.
One reason is that pretraining is not directly optimized for the supervised semantic segmentation objective of the target benchmark. As a result, the pretrained encoder may capture useful geometric structure and semantic regularities, but its features are not necessarily aligned with the precise class boundaries required by the downstream dataset, especially for long-tail categories and fine-grained object distinctions in point clouds with uneven density. In addition, different outdoor datasets may still exhibit nontrivial gaps in sensor characteristics, point distribution, point density, and annotation policy. Giving the model more capacity to adapt the representation to the target dataset often yields better performance.

[Garfield2005]

Got it, thanks for the response.