The final accuracy differs from what was reported in the paper

[ChenHuyoo]

I conducted experiments on Cityscale with configurations referencing sam_road. I noticed a significant gap in the recall metric compared to the paper's reported results, along with a slight difference in F1-score. Is this discrepancy expected, or could it be due to experimental configuration issues?
TOPO 0.7693445465917917 Precision 0.8946886302545709 Recall 0.6776406649652406
APLS: 67.34}

[Morefree303]

Thank you for your attention to our work. Since the length of the dataset for each epoch of Samroad is different from that of Sam-Road++, our epoch is set to 170 (which is the same steps as Samroad).

[ChenHuyoo]

Thank you for your attention to our work. Since the length of the dataset for each epoch of Samroad is different from that of Sam-Road++, our epoch is set to 170 (which is the same steps as Samroad).

Thank you for your reply. I further attempted to reproduce the results. Further experimental results show that the APLS metric differs by 1-2 percentage points, which seems within the normal range of fluctuation. However, the Recall and F1 score metrics still have a certain gap compared to those reported in the paper. Therefore, I would like to know if there are other configurations I might have overlooked?

[Morefree303]

Thank you for your question, I want to know whether your threshold is obtained according to the test.py？I will upload my config training file later.

[ChenHuyoo]

Yes, I obtained three thresholds through testing and then performed the inference. I will check again after you upload the configuration file. Thank you.

[Morefree303]

This is the inference configuration of cityscle, maybe it will be helpful to you. We will upload the complete config file later BASE_LR: 0.001
BATCH_SIZE: 16
DATASET: cityscale
DATA_WORKER_NUM: 1
ENCODER_LORA: false
ENCODER_LR_FACTOR: 0.1
FOCAL_LOSS: false
FREEZE_ENCODER: false
INFER_BATCH_SIZE: 64
INFER_PATCHES_PER_EDGE: 16
ITSC_NMS_RADIUS: 8
ITSC_THRESHOLD: 0.242
MAX_NEIGHBOR_QUERIES: 16
NEIGHBOR_RADIUS: 64
NO_SAM: false
PATCH_SIZE: 512
ROAD_NMS_RADIUS: 16
ROAD_THRESHOLD: 0.404
SAMPLE_MARGIN: 35
SAM_CKPT_PATH: ../sam_ckpts/sam_vit_b_01ec64.pth
SAM_VERSION: vit_b
TOPONET_VERSION: normal
TOPO_SAMPLE_NUM: 512
TOPO_THRESHOLD: 0.443
TRAIN_EPOCHS: 170
USE_SAM_DECODER: false

[ChenHuyoo]

Thanks for the info, could you please continue to provide the Spacenet configuration info here?

[hhd161]

感谢您提供的信息，您能否继续在此处提供 Spacenet 配置信息？

我想知道您是怎么生成最后的评估结果的，期待您的回复！

[ChenHuyoo]

感谢您提供的信息，您能否继续在此处提供 Spacenet 配置信息？

我想知道您是怎么生成最后的评估结果的，期待您的回复！

It's possible that the configuration is different from the author's, which could explain the discrepancy in the results. I am also waiting for the author's reply.

[hhd161]

感谢您提供的信息，您能否继续在此处提供 Spacenet 配置信息？

我想知道您是怎么生成最后的评估结果的，期待您的回复！

It's possible that the configuration is different from the author's, which could explain the discrepancy in the results. I am also waiting for the author's reply.

这个topo评价指标的代码我一直搞不出来，你能指导我一下吗，我将不胜感激！

[prusswan]

My results after epoch 112 (this is best by val_loss):

TOPO 0.7987529605814887 Precision 0.8853544001728818 Recall 0.7309169330284802
APLS 0.6695399999999999

Not too far from the reported metrics:
80.01 88.39 73.39 68.34

metric progression:

[ChenHuyoo]

My results after epoch 112 (this is best by val_loss):

TOPO 0.7987529605814887 Precision 0.8853544001728818 Recall 0.7309169330284802 APLS 0.6695399999999999

Not too far from the reported metrics: 80.01 88.39 73.39 68.34

metric progression:

After setting SAMPLE_MARGIN to 35, I obtained results on the CityScale dataset that closely matched those in the paper. However, have you tried running it on the Spacenet dataset? Since the authors didn't disclose their configuration, my results are still significantly different at this point.

[prusswan]

After setting SAMPLE_MARGIN to 35, I obtained results on the CityScale dataset that closely matched those in the paper. However, have you tried running it on the Spacenet dataset? Since the authors didn't disclose their configuration, my results are still significantly different at this point.

I have yet to try on spacenet, but you can try this config. Also, this seems to be a successor project.

My intent was to verify the cityscale results before trying to work on the larger globalscale data, so just the cityscale baseline is good enough for me.