HANG dr_app_start() from a non-main thread hangs up

[suyashmahar]

Describe the bug
If I call dr_app_start() from a non-main thread [2], the call hangs up. I do see the drmemtrace outdir, but that is it.
However, if I call dr_app_start() from the main thread [1], drmemtrace works correctly.

To Reproduce

Main thread calls dr_app_start() [1]

1. Setup the environment:

# export DR_ROOT=/home/smahar/DynamoRIO-Linux-11.90.20391
# export DYNAMORIO_OPTIONS="-client_lib $DR_ROOT/tools/lib64/release/libdrmemtrace.so;;-offline -code_api"
# export LD_LIBRARY_PATH="$DR_ROOT/tools/lib64/release/;$DR_ROOT/lib64/release/"

2. Build and run the application that calls dr_app_start() from the main thread:

# make -C ~/pi
g++ -O3 -march=native -mavx2 -mavx512f -mfma -L/home/smahar/DynamoRIO-Linux-11.90.20391/lib64/release/ pi.cc -o pi -ldynamorio
# ~/pi/pi 10000

Works, generates drmemtrace.pi.XXX.XXX.dir:

# ls -lah | grep drmemtrace 
drwxr-x---  3 smahar users    0 Nov  2 12:25 drmemtrace.pi.2698458.8781.dir
# du -sh drmemtrace.pi.2698458.8781.dir     
1.3G    drmemtrace.pi.2698458.8781.dir

Calling dr_app_start() from non-main thread [2] doesn't work:

[Environment unchanged]
Build and run the application that calls dr_app_start() from a non-main thread:

# make -C ~/pi
g++ -O3 -march=native -mavx2 -mavx512f -mfma -L/home/smahar/DynamoRIO-Linux-11.90.20391/lib64/release/ pi.cc -o pi -ldynamorio

After running for a minute, drmemtrace directory is empty and the log file does not have any output other than what was written before calling dr_app_start():

# cat /tmp/api_calls.txt
Tracing
# ls -lah | grep drmemtrace
drwxr-x---  3 smahar users    0 Nov  2 12:29 drmemtrace.pi.2767512.9381.dir
# du -sh drmemtrace.pi.2767512.9381.dir                             
0       drmemtrace.pi.2767512.9381.dir

Expected behavior
dr_app_start() call should return and trace the application.

Screenshots or Pasted Text
If applicable, add screenshots to help explain your problem. For text, please cut and paste the text here, delimited by lines consisting of three backtics to render it verbatim, like this:

Code

The only difference in these two programs is how the dr_app_start() and dr_app_stop() is called.

[1]

#include <algorithm>
#include <cmath>
#include <iostream>
#include <random>
#include <vector>
#include <cstdlib>
#include <unistd.h>
#include <thread>
#include <fstream>
#include <cassert>


// DynamoRIO required
#define LINUX
#if defined(__x86_64__) || defined(__i386__)
#define X86_64
#elif defined(__aarch64__) || defined(__arm__)
#define ARM_64
#endif

#include "/home/smahar/DynamoRIO-Linux-11.2.0/include/dr_api.h"

using point_t = std::pair<double, double>;

std::vector<point_t> generateRandomPointsInSquare(double length, size_t nbPoints) {
    std::random_device random_device;
    auto engine = std::mt19937{random_device()};

    auto x_dist = std::uniform_real_distribution<double>{-length, length};
    auto y_dist = std::uniform_real_distribution<double>{-length, length};

    auto points = std::vector<point_t>(nbPoints);
    std::generate_n(points.begin(), nbPoints,
                    [&x_dist, &y_dist, &engine]() { return std::make_pair(x_dist(engine), y_dist(engine)); });

    return points;
}

bool inCircle(const point_t& pt, double radius) {
    return pt.first * pt.first + pt.second * pt.second <= radius * radius;
}

double estimatePi(int n, int m) {
    const auto radius = std::pow(10.0, n);

    const auto nbPoints = static_cast<size_t>(std::pow(10, m));
    const auto points = generateRandomPointsInSquare(radius, nbPoints);
    const auto nbPointsInCircle =
        std::count_if(points.cbegin(), points.cend(), [radius](const point_t& pt) { return inCircle(pt, radius); });

    return 4 * static_cast<double>(nbPointsInCircle) / static_cast<double>(nbPoints);
}

double computeError(double estimate) {
    constexpr auto PI = double{3.14159265359};
    return std::abs(estimate - PI);
}

void TracingThread() {
    // open to write to the end
  std::ofstream out("/tmp/api_calls.txt", std::ios_base::app);
  while (true) {
    out << "Tracing" << std::endl;
    dr_app_start();
    assert(dr_app_running_under_dynamorio());
    // out << "[inside] Tracing" << std::endl << std::flush;
    sleep(1);
    dr_app_stop();
    out << "Stopped" << std::endl;
    sleep(4);
    out << "sleep done" << std::endl << std::flush;
  }
}

int main(int argc, const char* argv[]) {
    if (argc != 2) {
        std::cerr << "./pi <iters>" << std::endl;
        exit(1);
    }

    dr_app_setup();

    // auto tracing_thread = std::thread(TracingThread);
    // tracing_thread.detach();

    for (auto n = 0; n <= (int)atoi(argv[1]); ++n) {
        if (n%10 == 0)
            dr_app_start();

        for (auto m = 0; m <= 7; ++m) {
            std::cout << computeError(estimatePi(n, m)) << ' ';
        }
        if (n%10 == 0)
            dr_app_stop();
        std::cout << '\n';
    }

    return 0;
}

[2]

#include <algorithm>
#include <cmath>
#include <iostream>
#include <random>
#include <vector>
#include <cstdlib>
#include <unistd.h>
#include <thread>
#include <fstream>
#include <cassert>


// DynamoRIO required
#define LINUX
#if defined(__x86_64__) || defined(__i386__)
#define X86_64
#elif defined(__aarch64__) || defined(__arm__)
#define ARM_64
#endif

#include "/home/smahar/DynamoRIO-Linux-11.2.0/include/dr_api.h"

using point_t = std::pair<double, double>;

std::vector<point_t> generateRandomPointsInSquare(double length, size_t nbPoints) {
    std::random_device random_device;
    auto engine = std::mt19937{random_device()};

    auto x_dist = std::uniform_real_distribution<double>{-length, length};
    auto y_dist = std::uniform_real_distribution<double>{-length, length};

    auto points = std::vector<point_t>(nbPoints);
    std::generate_n(points.begin(), nbPoints,
                    [&x_dist, &y_dist, &engine]() { return std::make_pair(x_dist(engine), y_dist(engine)); });

    return points;
}

bool inCircle(const point_t& pt, double radius) {
    return pt.first * pt.first + pt.second * pt.second <= radius * radius;
}

double estimatePi(int n, int m) {
    const auto radius = std::pow(10.0, n);

    const auto nbPoints = static_cast<size_t>(std::pow(10, m));
    const auto points = generateRandomPointsInSquare(radius, nbPoints);
    const auto nbPointsInCircle =
        std::count_if(points.cbegin(), points.cend(), [radius](const point_t& pt) { return inCircle(pt, radius); });

    return 4 * static_cast<double>(nbPointsInCircle) / static_cast<double>(nbPoints);
}

double computeError(double estimate) {
    constexpr auto PI = double{3.14159265359};
    return std::abs(estimate - PI);
}

void TracingThread() {
    // open to write to the end
  std::ofstream out("/tmp/api_calls.txt", std::ios_base::app);
  while (true) {
    out << "Tracing" << std::endl;
    dr_app_start();
    assert(dr_app_running_under_dynamorio());
    out << "Inside tracing" << std::endl;
    sleep(1);
    dr_app_stop();
    out << "Iteration complete" << std::endl;
    sleep(4);
  }
}

int main(int argc, const char* argv[]) {
    if (argc != 2) {
        std::cerr << "./pi <iters>" << std::endl;
        exit(1);
    }

    dr_app_setup();

    auto tracing_thread = std::thread(TracingThread);
    tracing_thread.detach();

    for (auto n = 0; n <= (int)atoi(argv[1]); ++n) {
        // if (n%10 == 0)
        //     dr_app_start();

        for (auto m = 0; m <= 7; ++m) {
            std::cout << computeError(estimatePi(n, m)) << ' ';
        }
        // if (n%10 == 0)
        //     dr_app_stop();
        std::cout << '\n';
    }

    return 0;
}

Versions

• What version of DynamoRIO are you using? 11.90.20391
• Does the latest build from https://github.com/DynamoRIO/dynamorio/releases solve the problem? No
• What operating system version are you running on? ("Windows 10" is not sufficient: give the release number.) CentOS 9. Linux v6.9.0
• Is your application 32-bit or 64-bit? 64

Additional context
Add any other context about the problem here.

[suyashmahar]

Update: Tracing works correctly if I call dr_app_setup() from the same thread as dr_app_start(). However, this seem to contradict the docs for dr_app_setup():

Application-wide initialization. Must be called before any other API function. Returns zero on success.

[abhinav92003]

Thanks for tracking down the issue. Yes, dr_app_setup and dr_app_start should be invoked by the same thread. We do have dr_app_setup_and_start which may be easier for some use cases, but there's also #1305 which notes the need for separate setup and start APIs.

However, this seem to contradict the docs for dr_app_setup():

Not sure how the docs are contradictory. We can surely augment them to clarify that the setup and start should be in the same thread. Please feel free to send a PR to improve the documentation if you have suggestions.

[suyashmahar]

Thank you for confirming that they should be called from the same thread.

[suyashmahar]

Created this: #7709