Very inefficient with memory on CPU only mode.

[Elderx]

I have 20 cores of CPU available on Arch Linux with 16G of RAM. I can only utilize 4 cores, if i bump it to for example 8 cores, oom killer will very fast kill the process. Also seems like we currently have absolutely no way of knowing how long things take. Would be good to show some output for example:
a) how long does processing of 1 image takes
b) total progress so far

Currently this is everything we get when moving into "Sift extraction" phase. User is left guessing how long it might take.

`command_out: Processing 390 images on 19 threads...
Successfully resized 390 images.

Threads: 4`

[edin45]

Hey there!

So for the memory inefficiency in cpu only - I cant really do anything there - thats more a colmap thing.

As for progress - that is a good call I will add a Logs windows that streams the command outputs, which can be toggled open.

[Elderx]

From other issues comments: "so I will just removed them & remove ram monitoring all together". We still should not disable basic stability features. We could also dynamically calculate the optimal core count based on ram used/free on the system.

[edin45]

I mean yeah thats fair - I'll add dynamic core count calculation for the colmap steps & leave uncapped for openmvs (unless it is set in settings).

but ram monitoring I will not add back in, as all it is is if memory gets full, the app kills all sub processes - so its not actually going to improve memory usage.

[Elderx]

Hmm, would it also be possible to optimize core count during the process. Since after the first Sift Extraction step, the next step is Sift Matching, and this step seems to use much less memory --> at this point we could bump the core count up without getting memory limited. Since if this step is basically another colmap command in the background, we could give it different core count with flags before running it ?

[Elderx]

I mean yeah thats fair - I'll add dynamic core count calculation for the colmap steps & leave uncapped for openmvs (unless it is set in settings).

but ram monitoring I will not add back in, as all it is is if memory gets full, the app kills all sub processes - so its not actually going to improve memory usage.

At least on Arch Linux, the app itself did not grafecully kill its own subprocesses before oomkiller just took down everything parented to that process.

[edin45]

Yeah I could see something where we dynamically calculate the safe core count in the memory heavy steps & then just uncap it on the lighter ones.

With the ram watcher... yeah it was broken on linux, and caused false positive malware detection on windows so... I may try to bring back a better version in the future as I do see it being useful...

[edin45]

Ok so I released v1.1.6 just now & there we do a safe core count estimation for the sift extraction step only so far: https://github.com/edin45/simple_photogrammetry_gui/releases/tag/v1.1.6

let me know how you like it / if it is good now (or if it still takes too much ram, then I can tweak it a bit more as well)