questions about rootba on visual odometry

[empty-spacebar]

Hello, thanks for open-sourcing your great work.

Here are my questions:

1, Why did you implement rootBA on Basalt rather than DSO, another famous work from TUM? Since rootBA is designed for solving large-scale BA, and DSO maintains a much larger BA.

2, Actually I implemented rootBA on a direct visual odometry, BUT it ended up with about 7% more CPU occupation, worse accuracy and robustness. HOWEVER basalt shows remarkable performance no matter on runtime or accuracy. Is it the answer of question 1?

3, Does Basalt's accuracy benefit from stereographic projection rather than rootBA?

4, "...to keep the number of optimiziation variables small, we parametrize the bearing vector in 3D space using a minimal representation, which is two-dimensional", says the paper of Basalt. But actually the optimiziation dimension is still three. Could you please make a further explanation?

[NikolausDemmel]

1, Why did you implement rootBA on Basalt rather than DSO, another famous work from TUM? Since rootBA is designed for solving large-scale BA, and DSO maintains a much larger BA.

The technique we presented in the paper is suitable for different approaches. We just found it to be particularly advantageous for a setup like in Basalt. One can also try it in cases like DSO, but there may be some challenges, as you have found out. We didn't try it for this project.

2, Actually I implemented rootBA on a direct visual odometry, BUT it ended up with about 7% more CPU occupation, worse accuracy and robustness. HOWEVER basalt shows remarkable performance no matter on runtime or accuracy. Is it the answer of question 1?

Hard to say without further information. The DSO code is quite complex and not particularly well-documented. AFAIK it employs a lot of computational tricks to make things fast. It might be hard to have all of that in mind when making changes.

One thing to keep in mind is that in DSO every observation has an 8-dimensional residual, while in Basalt it's 2-dimensional. This means the Jacobians have 8 rows per observation (expect for the camera that hosts the point, which doesn't have a residual), while in Basalt it's only 2 rows per observation. Larger Jacobians also mean that the QR factorization is more expensive, so the runtime tradeoff might be different.

However, I would expect superior robustness / numerical stability also for DSO if you use nullspace marginalization compared to the Schur complement, in particular in single precision.

3, Does Basalt's accuracy benefit from stereographic projection rather than rootBA?

I'm not sure what you mean. The use of stereographic projection in the landmark parameterization is independent from the ideas of square root bundle adjustment / square root marginalization.

4, "...to keep the number of optimiziation variables small, we parametrize the bearing vector in 3D space using a minimal representation, which is two-dimensional", says the paper of Basalt. But actually the optimiziation dimension is still three. Could you please make a further explanation?

This just refers to 2-dimensions for the landmark direction. Alternative non-minimal parameterizations of the direction could be a bearing vector or a rotation matrix. Together with inverse depth it's still 3 dimensional, that is correct.

[empty-spacebar]

Thanks for your detailed reply.

After failing at replacing backend of DSO with rootBA, I came to the conclusion that direct mathod and rootBA do not match.

Then I found basalt is quiet unstable when camera holds still, see the follow clip:

b2caa872ed29b4049d7d6a45380e933b.mp4

Keypoints being filtered off by the second trackPoint(), aka KLT from target frame to current frame, causes the blue points dancing in the frame.

Do you have any idea about how to fix it?

[NikolausDemmel]

Keep in mind that Basalt is research software and is also not under development any more.

You could check out @mateosss's fork of Basalt, where he has made a lot of improvements to make it more robust (for XR applications). I believe he has added "SLAM features", i.e. persistent landmarks that should help with stability when stationary. You can find the fork here: https://gitlab.freedesktop.org/mateosss/basalt

[mateosss]

The instability you are observing in that video is due to the frame_to_frame optical flow, which makes features drift a bit after each frame.
You probably won't observe it if you change config.optical_flow_type to patch. In that case you always use the first observed patch of the feature for matching in the new image, which eliminates the drift. The downside is that you have shorter feature tracks overall.

The "feature recall" functionality in my fork, the "SLAM features" Nikolaus is referring to, help with this since it can use both the frame_to_frame feature tracking by default and fallback to patch tracking, or vice-versa depending on your needs. Since my fork's target application is XR, and the recall feature, by default, introduces annoying drift-correcting jumps in the trajectory that improve ATE but make you sick while wearing a headset, recall is disabled by default but you can toggle it and play around with it in the settings.

If you need help with that feel free to open an issue in that repo.

[empty-spacebar]

basalt_mateo_20250306-175742.mp4

Thanks for replies from both of you, the idea of Monado Basalt looks great.

But unfortunately, things didn't get better.

[mateosss]

Do you have a dataset/config/calibration file to share? I can give it a look.

[empty-spacebar]

Do you have a dataset/config/calibration file to share? I can give it a look.

https://drive.google.com/drive/folders/1eK_7y76h7beDM0a1Woh3F2UZkiw_c5pO?usp=drive_link

Try these files I just uploaded. Let me know if the link above isn't accessible.

[mateosss]

Indeed I cannot access the link, I requested permission.

[empty-spacebar]

Indeed I cannot access the link, I requested permission.

I turned off the access restriction, pleease try again

[mateosss]

I still don't have access, you should've received a request in your Gmail account.

[empty-spacebar]

I have accepted your request, try again?

[mateosss]

Well one quick thing you can do with your config file is the following for example:

<         "config.optical_flow_type": "frame_to_frame",
<         "config.optical_flow_detection_grid_size": 50,
<         "config.optical_flow_detection_num_points_cell": 1,
<         "config.optical_flow_detection_min_threshold": 5,
---
>         "config.optical_flow_type": "patch",
>         "config.optical_flow_detection_grid_size": 40,
>         "config.optical_flow_detection_num_points_cell": 2,
>         "config.optical_flow_detection_min_threshold": 10,

i.e., use "patch" tracking to avoid feature drift, increase the lowest quality of feature accepted, increase amount of features per cell to 2 (and reduce cell size to 40 to be a multiple of your resolution).

With this you get the following more stable result:

out2.mp4

There is still one source of annoyance, which is that every time a keyframe is marginalized in Basalt, all its hosted landmarks are removed for at least one frame from the "landmark database" and thus from the jacobian, and it shows as those set of features disappearing and repeating at the same time:

out3.mp4

One way of fixing this would be to have some kind of "landmark migration" procedure that would allow basalt to change the host of a landmark when a host keyframe is being marginalized.

[empty-spacebar]

Thanks for your detailed advice, increasing extracted points works but leads to more CPU loading, which is unacceptable for the SoC I use.

As for "landmark migration", intersting idea, I will take a try once I finish the project I currently work on.