Conclusion

The overall goal of this project was to evaluate the performance of feature matching algorithms in order to improve the 3D-Scans from the 3D-Scanner in the Pixelflux Laboratory at the Hochschule Karlsruhe. In order to do that, a closer look at classical feature matching algorithms was needed. Therefore, the architecture of todays most popular algorithm SIFT was evaluated and examined for possible points of failure when used on noisy images. Afterwards a research for Deep Feature Matching Algorithms was started, where the algorithm proposed in DFM: A Performance Baseline for Deep Feature Matching was found to be worth testing. Next the architecture of this paper was analyzed to determine, if it could be suitable for improving feature matching performance on noisy images. Since the deepest layers of the CNN promised robust matches, while filtering out noise, the next step was to evaluate the performance, by testing the algorithm with different noise types. Therefore, the IME repository had to be adapted to augment the dataset with noise, plot the results and execute the algorithms with the correct parameters. This proved to be very helpful for evaluation and can be built upon in the future to evaluate different algorithms.

However a final performance measurement was still missing, which is why the implementation of the DFM-algorithm into Meshroom was determined to be the final step to conclude this research. The three nodes DeepFeatureMatching, DFMImageTree and DeepFeatureMatchingAnalyzer were successfully implemented into Meshroom and can be used to generate features and matches. Unfortunately the photogrammetry pipeline fails with an error in the following StructureFromMotion node. Therefore, the DFM-Algorithm could not be fully tested in the application, that was initially proposed. At this point, further research is needed, which is why this documentation was written with a developer centric focus. It should be able to serve as a starting point for further investigation into Deep Feature Matching Algorithms.

Next steps

Solving the Structure from Motion Problem

The error logs of the structure from motion node can be found in a discussion, that I started to the meshroom forums. Here, someone suggested to take a look at the source code of the structure from motion node, specifically at the fuseMatchesIntoTracks function. A first look into the source code reveals, that SfM fails at the computePyramidPerView function, which could rely on the descriptors being present in the same directory as the features.

Therefore, it could be a first step, to extract the descriptors from the feature map of the first layer in the DFM-algorithm and save them into the same folder as the matches.

Testing additional Algorithms

Other feature matching algorithms, such as [LoFTR: Detector-Free Local Feature Matching with Transformers][loftr] could be tested for it's performance on noisy images using the IME repository.

Implementing additional Algorithms into Meshroom

The Developer Documentation should provide all necessary information to implement other algorithms into Meshroom.