Merge pull request #119 from lcmrl/add_new_local_features

Add new local features

Author: lcmrl

README.md

@@ -18,8 +18,6 @@
 
 Multivew matcher for SfM software. Support both deep-learning based and hand-crafted local features and matchers and export keypoints and matches directly in a COLMAP database or to Agisoft Metashape by importing the reconstruction in Bundler format. Now, it supports both OpenMVG and MicMac. Feel free to collaborate!
 
-While `dev` branch is more frequently updated, `master` is the default more stable branch and is updated from `dev` less frequently. If you are looking for the newest developments, please switch to `dev`.
-
 For how to use DIM, check the <a href="https://3dom-fbk.github.io/deep-image-matching/">Documentation</a> (updated for the master branch).
 
 **Please, note that `deep-image-matching` is under active development** and it is still in an experimental stage. If you find any bug, please open an issue. **For the licence of individual local features and matchers please refer to the authors' original projects**.
@@ -32,25 +30,47 @@ Key features:
 - Support for image rotations
 - Compatibility with several SfM software
 - Support image retrieval with deep-learning local features
-
-| Supported Extractors               | Supported Matchers                                        |
-| ---------------------------------- | --------------------------------------------------------- |
-| &check; SuperPoint                 | &check; Lightglue (with Superpoint, Disk, and ALIKED)     |
-| &check; DISK                       | &check; SuperGlue (with Superpoint)                       |
-| &check; Superpoint free            | &check; Nearest neighbor (with KORNIA Descriptor Matcher) |
-| &check; SRIF                       | &check; LoFTR (only GPU)                                  |
-| &check; ALIKED                     | &check; SE2-LoFTR (no tiling and only GPU)                |
-| &check; KeyNet + OriNet + HardNet8 | &check; RoMa                                              |
-| &check; DeDoDe (only GPU)          | &#x2610; GlueStick                                        |
-| &check; SIFT (from Opencv)         |
-| &check; ORB (from Opencv)          |
-
-| Supported SfM software                        |
-| --------------------------------------------- |
-| &check; COLMAP                                |
-| &check; OpenMVG                               |
-| &check; MICMAC                                |
-| &check; Agisoft Metashape                     |
+- Graph-based clustering
+- Run SfM directly in DIM (pycolmap, openmvg, etc)
+
+### Supported Extractors
+
+| Algorithm       | Year | Paper link   | Github link | Notes |
+| ---------       | ---- | -----------   | ---------- | ----- |
+| RIPE            | 2025 | [link](https://arxiv.org/abs/2507.04839) | [link](https://github.com/fraunhoferhhi/RIPE) | supported |
+| RDD sparse      | 2025 | [link](https://arxiv.org/abs/2505.08013) | [link](https://github.com/xtcpete/rdd) | supported |
+| LiftFeat        | 2025 | [link](https://www.arxiv.org/abs/2505.03422) | [link](https://github.com/lyp-deeplearning/LiftFeat) | supported |
+| XFeat           | 2024 | [link](https://arxiv.org/abs/2404.19174) | [link](https://github.com/verlab/accelerated_features) | supported |
+| DeDoDe          | 2024 | [link](https://arxiv.org/abs/2308.08479) | [link](https://github.com/Parskatt/DeDoDe) | only GPU  |
+| ALIKED          | 2023 | [link](https://arxiv.org/pdf/2304.03608) | [link](https://github.com/Shiaoming/ALIKED) | supported |
+| SRIF            | 2023 | [link](https://www.sciencedirect.com/science/article/abs/pii/S0924271623002277) | [link](https://github.com/LJY-RS/SRIF) | supported |
+| DISK            | 2020 | [link](https://arxiv.org/abs/2006.13566) | [link](https://github.com/cvlab-epfl/disk) | supported |
+| KeyNet          | 2019 | [link](https://arxiv.org/abs/1904.00889) | [link](https://github.com/axelBarroso/Key.Net) | supported |
+| SuperPoint      | 2018 | [link](https://arxiv.org/abs/1712.07629) | [link](https://github.com/magicleap/SuperPointPretrainedNetwork) | supported |
+| Superpoint open | 2018 | [link](https://arxiv.org/abs/1712.07629) | [link](https://github.com/rpautrat/SuperPoint) | supported |
+| HardNet      | 2017 | [link](https://arxiv.org/abs/1705.10872) | [link](https://github.com/DagnyT/hardnet) | supported |
+| ORB             | 2011 | [link](https://docs.opencv.org/3.4/d1/d89/tutorial_py_orb.html) | [link](https://ieeexplore.ieee.org/abstract/document/6126544) | from OpenCV |
+| SIFT            | 2004 | [link](https://docs.opencv.org/4.x/da/df5/tutorial_py_sift_intro.html) | [link](https://www.cs.ubc.ca/~lsigal/425_2024W1/ijcv04.pdf) | from OpenCV |
+
+
+### Supported Matchers
+
+| Algorithm | Year | Paper link | Github link | Notes |
+| --------- | ---- | ----------- | ---------- | ----- |
+| LightGlue | 2023 | [link](https://arxiv.org/pdf/2306.13643) | [link](https://github.com/cvg/LightGlue) | with SuperPoint, DISK, and ALIKED |
+| LighterGlue | 2023 | [link](https://arxiv.org/pdf/2306.13643) | [link](https://github.com/cvg/LightGlue) | with XFeat |
+| RoMa | 2023 | [link](https://arxiv.org/abs/2305.15404) | [link](https://github.com/Parskatt/RoMa) | supported |
+| SE2-LoFTR | 2022 | [link](https://openaccess.thecvf.com/content/CVPR2022W/IMW/papers/Bokman_A_Case_for_Using_Rotation_Invariant_Features_in_State_of_CVPRW_2022_paper.pdf) | [link](https://github.com/georg-bn/se2-loftr) | no tiling and only GPU |
+| LoFTR | 2021 | [link](https://arxiv.org/abs/2104.00680) | [link](https://github.com/zju3dv/LoFTR) | only GPU |
+| SuperGlue | 2020 | [link](https://arxiv.org/abs/1911.11763) | [link](https://github.com/magicleap/SuperGluePretrainedNetwork) | with SuperPoint |
+| Nearest Neighbor | - | - | - | from KORNIA |
+
+### Supported SfM software  
+
+| &check; [COLMAP](https://github.com/colmap/colmap)                                |
+| &check; [OpenMVG](https://github.com/openMVG/openMVG)                               |
+| &check; [MICMAC](https://github.com/micmacIGN/micmac)                                |
+| &check; [Agisoft Metashape](https://www.agisoft.com/)                     |
 | &check; Software that supports bundler format |
 
 ## Colab demo and notebooks
@@ -189,13 +209,14 @@ python ./join_databases.py --input path/to/dir/with/databases --output path/to/o
 
 ### Exporting the solution to Metashape
 
-To export the solution to Metashape, you can export the COLMAP database to Bundler format and then import it into Metashape.
-This can be done from Metashape GUI, by first importing the images and then use the function `Import Cameras` (File -> Import -> Import Cameras) to select Bundler file (e.g., bundler.out) and the image list file (e.g., bundler_list.txt).
+Suggested solution:
+* It is now possible to run SfM directly in Metashape using 2D observations extracted in DIM. You can use the script `export_to_bundler.py` from the scripts folder. It will create a fake bundler file. Then in Metashape import all the images you need, import camera poses using the bundler file, select all images and reset the alignment. Finally right click, align selected cameras (see [issue](https://github.com/3DOM-FBK/deep-image-matching/issues/94)).
+
+
+Other solutions:
+* To export the solution to Metashape, you can export the COLMAP database to Bundler format and then import it into Metashape. This can be done from Metashape GUI, by first importing the images and then use the function `Import Cameras` (File -> Import -> Import Cameras) to select Bundler file (e.g., bundler.out) and the image list file (e.g., bundler_list.txt).
 
-Alternatevely, you can use the `export_to_metashape.py` script to automatically create a Metashape project from a reconstruction saved in Bundler format.
-The script `export_to_metashape.py` takes as input the solution in Bundler format and the images and it exports the solution to Metashape.
-It requires to install Metashape as a Python module in your environment and to have a valid license.
-Please, refer to the instructions at [https://github.com/franioli/metashape](https://github.com/franioli/metashape).
+* Alternatevely, you can use the `export_to_metashape.py` script to automatically create a Metashape project from a reconstruction saved in Bundler format. The script `export_to_metashape.py` takes as input the solution in Bundler format and the images and it exports the solution to Metashape. It requires to install Metashape as a Python module in your environment and to have a valid license. Please, refer to the instructions at [https://github.com/franioli/metashape](https://github.com/franioli/metashape).
 
 ## How to contribute
 

notes.md

@@ -8,9 +8,6 @@
 - [ ] Testing on very large datasets ([Issue [#29](https://github.com/3DOM-FBK/deep-image-matching/issues/29)])
 - [ ] Use Github submodules instead of copying thirdpary code inside the repo
 - [ ] Add subpixel refinement of the matches (e.g., cross-correlation or [pixel-perfect-sfm](https://github.com/cvg/pixel-perfect-sfm))
-- [ ] Make semi-dense matcher work with multi-camera (Issue [[#24](https://github.com/3DOM-FBK/deep-image-matching/issues/24)])
-- [ ] Improve usage of multiple descriptors together
-- [ ] Finish extending compatibility to OpenMVG
 
 ## Bugs and Issues
 
@@ -26,7 +23,6 @@
 ## Other enhancements
 
 - [ ] Improve configuration management [_Hydra_](https://hydra.cc/docs/tutorials/structured_config/schema/) to make using yaml files, command line and GUI (Issue [[#48](https://github.com/3DOM-FBK/deep-image-matching/issues/48)])
-- [ ] Tests on satellite images
 - [ ] Add steerers + DeDoDe
 - [ ] Add Silk features
 - [ ] Add SIFT + LightGlue
@@ -59,3 +55,7 @@
 - [x] Add tests, documentation and examples (e.g. colab, ..)
 - [x] Cleanup repository to removed large files from Git history
 - [x] Update README CLI options
+- [x] Make semi-dense matcher work with multi-camera (Issue [[#24](https://github.com/3DOM-FBK/deep-image-matching/issues/24)])
+- [x] Improve usage of multiple descriptors together
+- [x] Finish extending compatibility to OpenMVG
+- [x] Tests on satellite images

src/deep_image_matching/config.py

@@ -157,6 +157,22 @@
         },
         "matcher": {"name": "kornia_matcher", "match_mode": "smnn", "th": 0.95},
     },
+    "liftfeat+kornia_matcher": {
+        "extractor": {
+            "name": "liftfeat",
+            "max_keypoints": 4096,
+            "detect_threshold": 0.05,
+        },
+        "matcher": {"name": "kornia_matcher", "match_mode": "smnn", "th": 0.99},
+    },
+    "ripe+kornia_matcher": {
+        "extractor": {
+            "name": "ripe",
+            "max_keypoints": 4096,
+            "detect_threshold": 0.5,
+        },
+        "matcher": {"name": "kornia_matcher", "match_mode": "smnn", "th": 0.95},
+    },
     "disk+lightglue": {
         "extractor": {
             "name": "disk",
@@ -169,6 +185,15 @@
             "name": "lightglue",
         },
     },
+    "xfeat+lighterglue": {
+        "extractor": {
+            "name": "xfeat",
+            "max_num_keypoints": 4096,
+        },
+        "matcher": {
+            "name": "lighterglue",
+        },
+    },
     "aliked+lightglue": {
         "extractor": {
             "name": "aliked",
@@ -185,6 +210,21 @@
             "filter_threshold": 0.1,  # match threshold
         },
     },
+    "rdd_sparse+lightglue": {
+        "extractor": {
+            "name": "rdd_sparse",
+            "max_num_keypoints": 4000,
+        },
+        "matcher": {
+            "name": "lightglue",
+            "n_layers": 9,
+            "depth_confidence": 0.95,  # early stopping, disable with -1
+            "width_confidence": 0.99,  # point pruning, disable with -1
+            "filter_threshold": 0.1,  # match threshold
+            "input_dim": 256,  # RDD descriptor dimension
+            "weights": '../../rdd/RDD/weights/RDD_lg-v2.pth',  # path to the weights
+        },
+    },
     "orb+kornia_matcher": {
         "extractor": {
             "name": "orb",
@@ -267,6 +307,10 @@
         "orb",
         "sift",
         "no_extractor",
+        "rdd_sparse",
+        "liftfeat",
+        "ripe",
+        "xfeat",
     ],
     "matchers": [
         "superglue",
@@ -277,6 +321,7 @@
         "adalam",
         "kornia_matcher",
         "roma",
+        "lighterglue",
     ],
     "retrieval": ["netvlad", "openibl", "cosplace", "dir"],
     "matching_strategy": [

src/deep_image_matching/extractors/liftfeat.py

@@ -0,0 +1,76 @@
+import yaml
+import numpy as np
+import torch
+from pathlib import Path
+
+from ..thirdparty.liftfeat.models.liftfeat_wrapper import MODEL_PATH, LiftFeat
+from .extractor_base import ExtractorBase
+
+
+class LiftFeatExtractor(ExtractorBase):
+    _default_conf = {
+        "name": "liftfeat",
+        "max_keypoints": 4000,
+        "detect_threshold": 0.05,
+    }
+    required_inputs = []
+    grayscale = False
+    descriptor_size = 128
+    
+
+    def __init__(self, config: dict):
+        # Init the base class
+        super().__init__(config)
+
+        # Load extractor
+        cfg = self.config.get("extractor")
+        detct_threshold = cfg.get("detect_threshold", self._default_conf["detect_threshold"])
+
+        self._extractor = LiftFeat(weight=MODEL_PATH,detect_threshold=detct_threshold)
+        self.max_num_keypoints = cfg.get("max_keypoints", self._default_conf["max_keypoints"])
+
+    @torch.no_grad()
+    def _extract(self, image: np.ndarray) -> np.ndarray:
+        # Extract features using LiftFeat's extract method (expects numpy array)
+        feats = self._extractor.extract(image)
+
+        # Convert tensors to numpy arrays
+        feats = {k: v.cpu().numpy() for k, v in feats.items()}
+
+        # Keep only the best keypoints based on scores
+        scores = feats["scores"]
+        if len(scores) > self.max_num_keypoints:
+            # Get indices of top max_num_keypoints by score
+            top_indices = np.argsort(scores)[::-1][:self.max_num_keypoints]
+            feats["keypoints"] = feats["keypoints"][top_indices, :]
+            feats["descriptors"] = feats["descriptors"][top_indices, :]
+            feats["scores"] = scores[top_indices]
+        
+        feats["descriptors"] = feats["descriptors"].T
+
+        return feats
+
+    def _frame2tensor(self, image: np.ndarray, device: str = "cuda"):
+        """
+        Convert a frame to a tensor.
+
+        Args:
+            image: The image to be converted
+            device: The device to convert to (defaults to 'cuda')
+        """
+        if len(image.shape) == 2:
+            image = image[None][None]
+        elif len(image.shape) == 3:
+            image = image.transpose(2, 0, 1)[None]
+        return torch.tensor(image / 255.0, dtype=torch.float).to(device)
+
+    def _rbd(self, data: dict) -> dict:
+        """Remove batch dimension from elements in data"""
+        return {
+            k: v[0] if isinstance(v, (torch.Tensor, np.ndarray, list)) else v
+            for k, v in data.items()
+        }
+
+
+if __name__ == "__main__":
+    pass

src/deep_image_matching/extractors/rdd_sparse.py

@@ -0,0 +1,75 @@
+import yaml
+import numpy as np
+import torch
+from pathlib import Path
+
+from ..thirdparty.rdd.RDD import RDD
+from .extractor_base import ExtractorBase
+
+
+class RDDSparseExtractor(ExtractorBase):
+    _default_conf = {
+        "name": "aliked",
+        "max_num_keypoints": 4000,
+    }
+    required_inputs = []
+    grayscale = False
+    descriptor_size = 128
+    
+
+    def __init__(self, config: dict):
+        # Init the base class
+        super().__init__(config)
+
+        # Load extractor
+        cfg = self.config.get("extractor")
+        config_path = Path(__file__).parent.parent / 'thirdparty' / 'rdd' / 'configs' / 'default.yaml'
+        weights_path = Path(__file__).parent.parent / 'thirdparty' / 'rdd' / 'RDD' / 'weights' / 'RDD-v2.pth'
+        
+        with open(config_path, 'r') as f:
+            network_config = yaml.safe_load(f)
+        self._extractor = RDD.build(config=network_config, weights=str(weights_path))
+        self.max_num_keypoints = cfg.get("max_num_keypoints", self._default_conf["max_num_keypoints"])
+
+    @torch.no_grad()
+    def _extract(self, image: np.ndarray) -> np.ndarray:
+        image_ = self._frame2tensor(image, self._device)
+
+        # Extract features using RDD's extract method
+        feats_list = self._extractor.extract(image_)
+        
+        # Get the first batch element (batch size is 1)
+        feats = feats_list[0]
+
+        # Convert tensors to numpy arrays
+        feats = {k: v.cpu().numpy() for k, v in feats.items()}
+
+        feats["keypoints"] = feats["keypoints"][:self.max_num_keypoints, :]
+        feats["descriptors"] = feats["descriptors"][:self.max_num_keypoints, :]
+
+        return feats
+
+    def _frame2tensor(self, image: np.ndarray, device: str = "cuda"):
+        """
+        Convert a frame to a tensor.
+
+        Args:
+            image: The image to be converted
+            device: The device to convert to (defaults to 'cuda')
+        """
+        if len(image.shape) == 2:
+            image = image[None][None]
+        elif len(image.shape) == 3:
+            image = image.transpose(2, 0, 1)[None]
+        return torch.tensor(image / 255.0, dtype=torch.float).to(device)
+
+    def _rbd(self, data: dict) -> dict:
+        """Remove batch dimension from elements in data"""
+        return {
+            k: v[0] if isinstance(v, (torch.Tensor, np.ndarray, list)) else v
+            for k, v in data.items()
+        }
+
+
+if __name__ == "__main__":
+    pass