Unable to Reproduce Reported InLoc Dataset Test Scores

[HustBestCat]

I attempted to integrate roma_indoor (based on publicly available weights) into the hloc framework and tested it using pipeline_inloc.ipynb. The scores I obtained are 58.6 / 73.7 / 82.8 and 54.2 / 68.7 / 77.1, which are significantly lower than the reported scores in the paper: 60.6 / 79.3 / 89.9 and 66.4 / 83.2 / 87.8. Could you help me determine if my approach was incorrect or if I used the wrong parameters?

Here's what I did:

1. Created a roma matcher based on BaseModel

image0 = data["image0"]  # (1, 3, H1, W1)
image1 = data["image1"]  # (1, 3, H2, W2)
h1, w1 = image0.shape[-2:]
h2, w2 = image1.shape[-2:]
# Run Roma
with torch.no_grad():
    dense_matches, dense_certainty = self.net.match(image0, image1)  # Adapted match for tensor scale input
    sparse_matches, sparse_certainty = self.net.sample(
        dense_matches, dense_certainty, num=self.max_num_matches
    )  # Adapted sample to output sampled certainty scores
    kpts0, kpts1 = self.net.to_pixel_coordinates(sparse_matches, h1, w1, h2, w2)
# Return in HLoc format
return {
    "keypoints0": kpts0,
    "keypoints1": kpts1,
    "scores": sparse_certainty,
}

2. Registered Configuration in match_dense.py

"roma_indoor": {
    "output": "roma_indoor",
    "model": {"name": "roma_indoor"},
    "preprocessing": {
        "grayscale": false,  // Critical! Must be false to pass RGB
        "resize_max": 1344,
        "dfactor": 14,
    },
    "max_error": 1,
    "cell_size": 1,
},

3. Executed Matching and Localization in pipeline_inloc.ipynb

dense_conf = match_dense.confs["roma_indoor"]
features, matches = match_dense.main(dense_conf, loc_pairs, dataset, export_dir=outputs)
localize_inloc.main(
    dataset, loc_pairs, features, matches, results, skip_matches=20
)

RANSAC settings were configured as follows:

estimation_options.ransac.max_error = 50
estimation_options.ransac.confidence = 0.99999
estimation_options.ransac.min_inlier_ratio = 0.1

Could there be specific steps or parameter settings that led to this performance discrepancy? Looking forward to your response and guidance.
Additionally, by calling the following function visualization.visualize_loc(results, dataset, n=1, top_k_db=1, seed=2), I got the visualization results, which look reasonably good.

[Parskatt]

Iirc it matters quite a lot how you sample the points, Ill see if I can find my old eval pipeline.

[HustBestCat]

Iirc it matters quite a lot how you sample the points, Ill see if I can find my old eval pipeline.

Got it, thanks! Looking forward to it.