BUG: featureExtraction in GPU mode has undefined behavior on completion

PopSift started to work after #428 was merged, but one of the steps in the pipeline is not working properly yet.

Problem

Expected behavior:
aliceVision_featureExtraction finishes successfully.

Actual behavior:
aliceVision_featureExtraction finishes its chunk of work but it makes individual socks disappear from my laundry and hangs in the PopSift destructor instead of exiting. Sometimes it throws a Boost threading exception instead.

Steps to Reproduce

1. Extract features with GPGPU acceleration enabled (uncheck “Force CPU extraction” in Meshroom)
2. Wait for process to exit successfully

Versions

• AliceVision branch/version: develop a304bdd
• OS: openSUSE Tumbleweed
• C++ compiler: GCC 7.3.1+r261272 (NVCC prevents me from using GCC 8). Have been able to reproduce this with GCC 6 and the bundled library versions as well.
• NVIDIA GeForce GTX 960 (GM206), driver 390.67, CUDA SDK 9.1

[zvrba]

Can you run it in a debugger and post a stack trace for all threads? Threading exceptions and (most probably related) hangs tend to occur if a mutex or condition variable get destroyed while they're in use. Alternately, you can attach the debugger with gdb -p <pid> and then dump stack trace for all threads; something like thread apply all bt

Thread 6 (Thread 0x7f237e7fc700 (LWP 24588)):
#0  0x00007f23985c34dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2393eb968f in PopSift::extractDownloadLoop() () from /usr/lib64/libpopsift.so.1.0.0
#2  0x00007f239640a0fd in thread_proxy () at libs/thread/src/pthread/thread.cpp:175
#3  0x00007f23985bd554 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f23982f0ccf in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f237effd700 (LWP 24587)):
#0  0x00007f23985c34dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2393ebe80d in PopSift::uploadImages() () from /usr/lib64/libpopsift.so.1.0.0
#2  0x00007f239640a0fd in thread_proxy () at libs/thread/src/pthread/thread.cpp:175
#3  0x00007f23985bd554 in start_thread () from /lib64/libpthread.so.0
#4  0x00007f23982f0ccf in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f237f7fe700 (LWP 24586)):
#0  0x00007f23985c34dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2387270d2d in ?? () from /usr/lib64/libcuda.so.1
#2  0x00007f2387229394 in ?? () from /usr/lib64/libcuda.so.1
#3  0x00007f2387270110 in ?? () from /usr/lib64/libcuda.so.1
#4  0x00007f23985bd554 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f23982f0ccf in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f237ffff700 (LWP 24585)):
#0  0x00007f23982e62c9 in poll () from /lib64/libc.so.6
#1  0x00007f238726e69b in ?? () from /usr/lib64/libcuda.so.1
#2  0x00007f23872d3a6f in ?? () from /usr/lib64/libcuda.so.1
#3  0x00007f2387270110 in ?? () from /usr/lib64/libcuda.so.1
#4  0x00007f23985bd554 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f23982f0ccf in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f2388f0d700 (LWP 24575)):
#0  0x00007f23982f25df in accept4 () from /lib64/libc.so.6
#1  0x00007f238726f4a6 in ?? () from /usr/lib64/libcuda.so.1
#2  0x00007f2387263a3d in ?? () from /usr/lib64/libcuda.so.1
#3  0x00007f2387270110 in ?? () from /usr/lib64/libcuda.so.1
#4  0x00007f23985bd554 in start_thread () from /lib64/libpthread.so.0
#5  0x00007f23982f0ccf in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f239ab838c0 (LWP 24570)):
#0  0x00007f23985c2f9f in pthread_cond_destroy@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2393eb7a38 in PopSift::~PopSift() () from /usr/lib64/libpopsift.so.1.0.0
#2  0x00007f2399a38a71 in std::unique_ptr<PopSift, std::default_delete<PopSift> >::~unique_ptr() () from /usr/lib64/libaliceVision_feature.so.0
#3  0x00007f2398230d07 in __cxa_finalize () from /lib64/libc.so.6
#4  0x00007f2399a08023 in __do_global_dtors_aux () from /usr/lib64/libaliceVision_feature.so.0
#5  0x00007ffcce6ea810 in ?? ()
#6  0x00007f239a9db966 in _dl_fini () from /lib64/ld-linux-x86-64.so.2
Backtrace stopped: frame did not save the PC

[zvrba]

Yes, thread 1 is calling pthread_cond_destroy from Popsift destructor while other Popsift threads (5 and 6) are waiting on pthread_cond_wait. From the symptoms, I'm 99% sure that it's the same condvar. (You can confirm that by switching to each thread, selecting the stack frame just before the pthread_cond_* call and printing out the condvar address.) This is most definitely a bug.

EDIT: the documentation states the following: "Attempting to destroy a condition variable upon which other threads are currently blocked results in undefined behavior." (From http://pubs.opengroup.org/onlinepubs/009695399/functions/pthread_cond_destroy.html)

[vnarasimhan]

Looks like a call to "uninit" (https://github.com/alicevision/popsift/blob/0c7ce9db47f5ff8cb853eb35d8c27f56a1e94045/src/popsift/popsift.cpp#L122) prior to PopSift destruction could take care of this issue. Ideally, this would be done before destroying any PopSift instances .. like the one here:

AliceVision/src/aliceVision/feature/sift/ImageDescriber_SIFT_popSIFT.cpp

Line 95 in 8ee6513

_popSift.reset(new PopSift(config, popsift::Config::ExtractingMode, PopSift::FloatImages));

. The PopSift sample app calls uninit here: https://github.com/alicevision/popsift/blob/0c7ce9db47f5ff8cb853eb35d8c27f56a1e94045/src/application/main.cpp#L326

[vnarasimhan]

Lightly tested patch that solves this problem. Based off 8ee6513

popsift.patch.txt

[fabiencastan]

@griwodz What do you think?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[simogasp]

It should be now fixed by #807

…

On Mon, Sep 14, 2020, 02:18 github-actions[bot] ***@***.***> wrote: This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#437 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKFEIGJOO6PEBHH7BCNASTSFVOMPANCNFSM4FIZT7FQ> .