rust driver_manager: initializing the driver creates some dangling resources which hangs application during shutdown

[Pranav2612000]

What happened?

Here's a sample test

  #[test]
  fn test_driver_setup() {
    const SNOWFLAKE_DRIVER_NAME: &str = "snowflake";
    let mut driver = ManagedDriver::load_from_name(
      SNOWFLAKE_DRIVER_NAME,
      None,
      AdbcVersion::default(),
      LOAD_FLAG_DEFAULT,
      None,
    )
    .unwrap();
    println!("Driver loaded");
  }

Here's the output on running the test

    Finished `test` profile [unoptimized + debuginfo] target(s) in 11.79s
     Running unittests src/lib.rs (target/debug/deps/<redacted>)

running 1 test
Driver loaded

As you see, the test never completes. Debugging further, I've pinpointed the issue to the ADBC FFI init function ( FFI_AdbcDriverInitFunc ) ( here )

On the same machine, the following code works as expected, with a clean shutdown

Stack Trace

How can we reproduce the bug?

Sample test case shared above

Environment/Setup

rust version -> rustc 1.86.0 (05f9846f8 2025-03-31)

Cargo.toml dependencies

adbc_core = "0.21.0"
adbc_driver_manager = "0.21.0"
arrow = { version = "57.0.0", features = ["prettyprint"] }
arrow-array = "57.0.0"
async-trait = "0.1.89"
dhat = { version = "0.3.3", optional = true }
tikv-jemallocator = "0.6.0"
tokio = { version = "1.45.1", features = ["full", "test-util"] }
tokio-executor = "0.1.10"
tokio-executor-trait = "2.1.1"
tokio-reactor-trait = "2.0.0"
tokio-retry = "0.3.0"
tokio-stream = "0.1.16"

build.rs contains script to generate gRPC code and linking for GPU stuff

Snowflake driver downloaded using dbc

[lidavidm]

Do you have a stack trace?

[Pranav2612000]

Nope. There's no error, so I don't see the trace by default. How do I generate one? Which one (at which line) should I share?

[Pranav2612000]

I see this error in the console, could it be related?

[lidavidm]

If you can use gdb/lldb/another debugger, you should be able to generate a backtrace. That message seems like your process didn't hang, but crashed.

[lidavidm]

If your application is hanging, then presumably you can generate a backtrace showing where it was stuck.

[Pranav2612000]

Here's the stack trace when the program hangs [ IMG ] ( Stack trace of the main thread, I see multiple threads running [ IMG ] )

Although the "console" window shows an error, the "Terminal" output does not display this error and just hangs.

[lidavidm]

(1) are you sure your IDE isn't automatically entering the debugger on a crash and causing your application to "hang"?
(2) this seems to be related to thread-locals, are you using those?
(3) how were you able to conclude this is related to ADBC?

[lidavidm]

(Also, it may be worth looking at all the threads to make sure you know which thread is stuck/segfaulted?)

[Pranav2612000]

(1) are you sure your IDE isn't automatically entering the debugger on a crash and causing your application to "hang"?

I guess the EXEC_BAD_ACCESS error we see is due to the IDE. In the stuck state, pressing the "Continue program" button prints this message in the console. I'm sure the IDE is not causing the "hang" because the behaviour is the same when I run it through ITerm

(2) this seems to be related to thread-locals, are you using those?

No. Although we do use tokio, I'm wondering if that could be causing this issue ( that's a difference between the example which works and the test which doesn't )

(3) how were you able to conclude this is related to ADBC?

I'm able to reproduce the issue with a very simple test ( shared above ). Commenting out the part which loads the driver works and the test exits. Using different driver ( tried with bigquery ) also has the same "stuck/hang" behaviour. Added sample panic! in the adbc codebase to debug the issue. Whenever the init FFI_AdbcDriverInitFunc runs, ( this line )the test goes into this state. ( Adding a panic immediately before this line doesn't cause a hang, but adding a panic immediately after causes the hang )

Let me look at all the threads once again, to see if I see anything interesting. By the way, Thank you for your help @lidavidm and the prompt replies 😄

[Pranav2612000]

When running with the load_from_name func commented, I see only 2 threads - the main thread and the test thread, which I think is the expected behaviour.

When running the load_from_name enabled, I see 8 threads on the last line of the test ( before drop ). 1 main thread ( the test runner ), 1 test thread and 6 other threads all having this sort of stack trace ( IMG ) ( These threads are probably created for connection management )

After the test completes, the main thread is stuck on __ulock_wait ( probably waiting for the test thread to complete ). The test thread has this trace

and all other threads are still stuck on psynch_cvwait ( although with a slightly different trace )

[Pranav2612000]

My hunch is that due to some reason, these 6 threads spawned could not be cleaned up correctly causing the test thread to hang which hangs the main thread and so the whole test. This is also the same behaviour I see when running the app.

[lidavidm]

Hmm. I'm curious if using the ADBC postgresql or sqlite drivers also shows this behavior, then. (Both Snowflake and BigQuery drivers are written in Go and so load the Go runtime.)

Can you provide a standalone reproducer? Is it literally that test, with no other dependencies? It doesn't appear to use tokio at all, yet you mention tokio makes a difference?

Are you able to inspect what pthread_tsd_cleanup is doing and try to see what kind of thing it is trying to cleanup?

[Pranav2612000]

Yup. You're right. postgresql and sqlite drivers both work. I don't see this "hang".

[Pranav2612000]

Can you provide a standalone reproducer? Is it literally that test, with no other dependencies? It doesn't appear to use tokio at all, yet you mention tokio makes a difference?

My bad. We do use tokio in the app, so I though it could be one reason, but that doesn't seem to be the case.
Here's a very simple test which also hangs -> https://github.com/Pranav2612000/snowflake-adbc-test/blob/8c62039796fefcc7fa64412d922355f83d89ac8c/src/main.rs#L59