Failing tests and deadlocks on linux/aarch64 (Amazon Graviton2)

[ulfworsoe]

Found on:

• Linux (Ubuntu 20.04.3) on aarch64 (Amazon Graviton2)
• TBB version 2021.4.0 and master (883c2e5)
• building with cmake+ninja (CC=clang CXX=clang++ cmake -GNinja -DCMAKE_INSTALL_PREFIX=$HOME/local/2021.4.0 -DCMAKE_BUILD_TYPE=Release).

Running the TBB tests, some tests failed:

• In master and 2021.4.0 test_composite_node appear to hang, killed manually after ~1000 seconds
• In 2021.4.0 test_concurrent_vector appear to hang, killed manually after ~1000 seconds
• test_eh_thead aborts

[erling-d-andersen]

More info in the duplicate but closed issue: #688

[odidev]

@anton-potapov

I have been working on installing and testing this package for amd64 and arm64 architectures. While testing I am getting 2 tests failure on my local arm server. It would be really helpful if you could share some pointers on it.

98% tests passed; 2 tests failed out of 133 
Total Test time (real) = 476.32 sec 
The following tests FAILED: 
         28 - test_resumable_tasks (SEGFAULT) 
         60 - test_eh_thread (Subprocess aborted) 
Errors while running CTest

Log for reference: oneTBB_tests_arm64.txt

[alexey-katranov]

test_eh_thread seems stange. It could not created 1024 and received an error from pthread_create but then oneTBB was able to create one more thread. Try to set a break point to pthread_create inside rml_thread_monitor.h and check why it does not fail.

As for other tests, it is quite difficult to suppose what is going wrong (supposedly, it might be related to relaxed memory model of aarch64). Is possible to share core dumps of hanged tests?

[erling-d-andersen]

Due to Christmas vacation we are not likely to return before January. Sorry.

[phprus]

@alexey-katranov, @anton-potapov Error in test_eh_thread - is not oneTBB issue.

Hardware: Raspberry Pi 4 8GB RAM.
OS:

1. Ubuntu 20.04.3 (kernel 5.4.0-1047-raspi) + docker, container: oraclelinux:8.5
2. Ubuntu 20.04.3 (kernel 5.4.0-1047-raspi) without any containers.

Test:

// g++ -pthread -std=c++17 pthread.cpp && ./a.out && echo OK || echo ERROR

#include <algorithm>
#include <atomic>
#include <condition_variable>
#include <thread>
#include <vector>
#include <iostream>

#include <sys/types.h>
#include <sys/time.h>
#include <sys/resource.h>
#include <unistd.h>


void limitThreads(size_t limit)
{
    rlimit rlim;

    int ret = getrlimit(RLIMIT_NPROC, &rlim);
    if (ret != 0)
    {
        std::cerr << "getrlimit has returned an error" << std::endl;
        exit(1);
    }

    rlim.rlim_cur = (rlim.rlim_max == (rlim_t)RLIM_INFINITY) ? limit : std::min(limit, rlim.rlim_max);

    ret = setrlimit(RLIMIT_NPROC, &rlim);
    if (ret != 0)
    {
        std::cerr << "setrlimit has returned an error" << std::endl;
        exit(1);
    }
}

static std::mutex m;
static std::condition_variable cv;
static std::atomic<bool> stop{ false };

static void* thread_routine(void*)
{
    std::unique_lock<std::mutex> lock(m);
    cv.wait(lock, [] { return stop == true; });
    return 0;
}

static void* new_thread_routine(void*)
{
    std::cerr << "sleep" << std::endl;
    sleep(60);
    return 0;
}

class Thread {
    pthread_t mHandle{};
    bool mValid{};
public:
    Thread() {
        mValid = false;
        pthread_attr_t attr;
        // Limit the stack size not to consume all virtual memory on 32 bit platforms.
        if (pthread_attr_init(&attr) == 0 && pthread_attr_setstacksize(&attr, 100*1024) == 0) {
            mValid = pthread_create(&mHandle, &attr, thread_routine, /* arg = */ nullptr) == 0;
        }
    }
    bool isValid() const { return mValid; }
    void join() {
        pthread_join(mHandle, nullptr);
    }
};


void check( int error_code, const char* routine )
{
    if (error_code)
    {
        std::cerr << routine << std::endl;
        _exit(1);
    }
}


int main()
{
    // Some systems set really big limit (e.g. >45К) for the number of processes/threads
    limitThreads(1024);

    std::thread /* isolate test */ ([] {
        std::vector<Thread> threads;
        stop = false;
        auto finalize = [&] {
            stop = true;
            cv.notify_all();
            for (auto& t : threads) {
                t.join();
            }
        };

        for (int i = 0;; ++i) {
            Thread thread;
            if (!thread.isValid()) {
                break;
            }
            threads.push_back(thread);
            if (i == 1024) {
                std::cerr << "setrlimit seems having no effect" << std::endl;
                finalize();
                return;
            }
        }

        pthread_t new_handle;
        pthread_attr_t s;
        check(pthread_attr_init( &s ), "pthread_attr_init has failed");
        pthread_t handle;
        void * arg = nullptr;

        int ec = pthread_create( &new_handle, &s, new_thread_routine, arg );
        if (ec) {
            std::cerr << "EXPECTED ERROR: " << "pthread_create has failed" << std::endl;
        } else {
            std::cerr << "UNEXPECTED OK: " << "pthread_create is not failed" << std::endl;
            _exit(1);
        }
        check( pthread_attr_destroy( &s ), "pthread_attr_destroy has failed" );
        pthread_join(new_handle, nullptr);

        finalize();
    }).join();

    return 0;
}

Output:

[root@ubuntu ~]# g++ -pthread -std=c++17 pthread.cpp
[root@ubuntu ~]# ./a.out && echo OK || echo ERROR
sleepUNEXPECTED OK: pthread_create is not failed
ERROR

Maybe bug in glibc or in ubuntu 20.04 kernel? ... Or in my test?

[alexey-katranov]

Maybe if (pthread_attr_init(&attr) == 0 && pthread_attr_setstacksize(&attr, 100*1024) == 0) failed (not pthread_create) when we tried to create 1024 threads?

[phprus]

@alexey-katranov YES!!!
pthread_attr_setstacksize returns EINVAL.

https://man7.org/linux/man-pages/man3/pthread_attr_setstacksize.3.html:

EINVAL The stack size is less than PTHREAD_STACK_MIN bytes.
On some systems, pthread_attr_setstacksize() can fail with the error EINVAL if stacksize is not a multiple of the system page size.

PTHREAD_STACK_MIN is present in <limits.h>

On aarch64 PTHREAD_STACK_MIN == 131072 or dynamic value: sysconf(_SC_THREAD_STACK_MIN).

[phprus]

@isaevil PR #697 fix only test_eh_thead test, but tests test_composite_node, test_concurrent_vector and test_resumable_tasks is not fixed by it.
Please, reopen this issue.

[isaevil]

@phprus I guess issue was closed automatically because you mentioned it. Reopening.