Log files of all models for all datasets

[Varamyr]

Hi,

first I would like to thank you for your very useful and interesting work, I've been relying on this work in order to validate my research on a new SSL method.
I'm writing because I'm having trouble reproducing the results claimed in the paper for some of the models and dataset.
It's possible that I've missed them but it would be very helpful if you would publish in this repo the log files related to the experiments of the models evaluated in this benchmark.

For now I've been testing on 5 datasets: MADOS, HLSBurnScars, SpaceNet7 (change detection), Sen1Floods11 and AI4SmallFarms.
I've encountered the most incosistent result in SpaceNet7cd and AI4SmallFarms.

Taking DOFA as reference I've obtained the following results in terms of IoU:

1. MADOS

Config used
task: segmentation
dataset: mados
encoder: dofa
decoder: seg_upernet
preprocessing: seg_focus_crop
criterion: cross_entropy
lr_scheduler: multi_step_lr
optimizer: adamw

• Paper: 59,58
• Retrained: 57,45

2. HLSBurnScars

Config used
task: segmentation
dataset: hlsburnscars
encoder: dofa
decoder: seg_upernet
preprocessing: seg_default
criterion: cross_entropy
lr_scheduler: multi_step_lr
optimizer: adamw

• Paper: 80,63
• Retrained: 86,038

3. SpaceNet7cd

Config used
task: change_detection
dataset: spacenet7cd
encoder: dofa
decoder: seg_siamupernet_conc
preprocessing: seg_default
criterion: dice
lr_scheduler: multi_step_lr
optimizer: adamw

• Paper: 61,84
• Retrained: 45,271

4. Sen1Floods11

Config used
task: segmentation
dataset: sen1floods11
encoder: dofa
decoder: seg_upernet
preprocessing: seg_default
criterion: cross_entropy
lr_scheduler: multi_step_lr
optimizer: adamw

• Paper: 89,37
• Retrained: 89,128

5. AI4SmallFarms

Config used
task: segmentation
dataset: ai4smallfarms
encoder: ssl4eo_mae_optical
decoder: seg_upernet
preprocessing: seg_default
criterion: cross_entropy
lr_scheduler: multi_step_lr
optimizer: adamw

• Paper: 27,07
• Retrained: 37,694 (here it looks better but with the following IoU per class Background 75.380, Crop Field 0.008)

Secondly I have some doubts about the finetuning strategy since in the paper, in Figure 4, the encoder is shown to be frozen for all modalities but in the code published the encoder is always finetuned along with decoder (which to me it would look also more intuitive for architectures like MAE).
Can you tell me whether the results published are obtained with a finetuned encoder or by training only the decoder?

[lauraset]

Hi, @Varamyr . Have you deal with the AI4SmallFarms, I also encounter the same thing:

The config is given below:

  - task: segmentation
  - dataset: ai4smallfarms
  - encoder: remoteclip
  - decoder: seg_upernet
  - preprocessing: seg_default
  - criterion: cross_entropy
  - lr_scheduler: multi_step_lr
  - optimizer: adamw
  - _self_