Evaluation of frozen features

[zhouh02]

Hello, I noticed your paper mentions experiments on the robustness of VGG10, ResNet50, and DINOv2 to changes in viewpoint and lighting. I'd like to ask about some experimental details: What datasets were VGG19 and ResNet pre-trained on? If possible, could you share your evaluation code? Thank you very much!
Wishing you success in your research!

[Parskatt]

Hi, I don't have the code currently (@davnords has been working on a better version), but it's basically the megadepth dense benchmark which is available in the codebase. VGG19 and ResNet are the default v1 weights in pytorch (imagenet 1k pretrained).

[davnords]

Hi @zhouh02. Will probably release something with @Parskatt that makes this benchmark more systematic in the future. For now, here is some rough code to do the same type of linear evaluation as in the table above. You can change the VisionEncoder for e.g. VGG19. The code depends on romatch being installed. The main entrypoint is benchmark.py.

Good luck!

linear-eval.zip

[zhouh02]

Thanks for sharing. I'll give it a try.

[zhouh02]

It appears that some files are missing from this linear-eval.zip file. Could you clarify whether this is due to confidentiality concerns or an accidental release of the wrong version?

[davnords]

Hi @zhouh02, I tried to only include the necessary files so might have forgotten some. What is the name of the missing files?

[zhouh02]

@davnords Thank you for taking the time to reply to my message so late at night. First and foremost, I want to express my gratitude to you! Regarding the ZIP file you previously uploaded, I noticed that the utils.py file is empty. Additionally, since my coding skills are somewhat limited, I find modifying this code challenging. If possible, I would appreciate it if you could share the complete code with me—including the full evaluation scripts adapted for both the VGG19 and DINOv2 backbone extraction networks. This would allow me to reproduce your results more efficiently and proceed with further research. I sincerely apologize for the inconvenience.

[davnords]

Nps, I'll give it a shot. I will also release something more straight forward in the future, so if you are in no rush you could wait for that :). I'll take a look at the code, one moment :)

[zhouh02]

I sincerely appreciate you taking the time to reply so late. I've been feeling quite rushed about this code lately, as I'm also trying to finish my first paper. ROMA is a paper I greatly admire and respect—I've gained many insights from it. I believe that in feature matching tasks, a robust feature extraction backbone is indeed crucial. Therefore, I'd greatly appreciate it if you could share the complete code for this frozen feature experiment. This would allow me to build upon your work and conduct further experiments more efficiently. I could then dedicate more time to testing new networks like DINOv3 or introducing additional metrics for a more comprehensive evaluation of which network performs better. (I realize my coding skills are limited, which is why I'm hesitant to ask this favor.) Once I have the results, I hope to share them with you for mutual learning and progress.

[davnords]

Here is some updated code. You can use main.py for VGG19 and DINOv2. I have not really been able to test the code in this format (I mainly copy and pasted from another repo I have), but it should work with some minor modifications such as import errors or something. Please tell me if you are missing any files, should be fine now I believe.

You need the megadepth dataset downloaded on the same format as is expressed in the readme here and which is more clearly described in the DKM repo

linear-eval.zip

[davnords]

Sounds good. Good luck on your paper. I am in no way affiliated with the RoMa work so all cred goes to the authors :).

Also, the package requirements should be similar to the general RoMa environment.

[davnords]

I am working on something slightly related, a feature backbone for matching. Happy to hear about your learnings.

/ David

[zhouh02]

@davnords Alright, I'll take a look now.