Regress tests stuck on macOS arm64

[qsliu2017]

What happens?

make installcheck might randomly stuck. setting duckdb.threads = 1 in test/regression/regression.conf could fix. So I guess it's a race condition.

upon stuck, the call stack looks as follows:

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x000000019e6a8c20 libsystem_malloc.dylib`xzm_malloc
    frame #1: 0x000000019e841a78 libc++abi.dylib`operator new(unsigned long) + 52
    frame #2: 0x000000010be4069c libduckdb.dylib`duckdb::PendingQueryResult::LockContext() + 56
    frame #3: 0x000000010be40a60 libduckdb.dylib`duckdb::PendingQueryResult::ExecuteTask() + 28
    frame #4: 0x000000010315348c pg_duckdb.dylib`ExecuteQuery(DuckdbScanState*) + 824
    frame #5: 0x0000000103153ad8 pg_duckdb.dylib`Duckdb_ExecCustomScan(CustomScanState*) + 112
    frame #6: 0x00000001028ecd94 postgres`standard_ExecutorRun + 288
    frame #7: 0x0000000102a96b98 postgres`PortalRunSelect + 220
    frame #8: 0x0000000102a96860 postgres`PortalRun + 472
    frame #9: 0x00000001028a6d7c postgres`ExecuteQuery + 388
    frame #10: 0x0000000102a97dec postgres`standard_ProcessUtility + 1468
    frame #11: 0x0000000103146200 pg_duckdb.dylib`DuckdbUtilityHook(PlannedStmt*, char const*, bool, ProcessUtilityContext, ParamListInfoData*, QueryEnvironment*, _DestReceiver*, QueryCompletion*) + 4252
    frame #12: 0x0000000102a97520 postgres`PortalRunUtility + 136
    frame #13: 0x0000000102a96aa8 postgres`FillPortalStore + 224
    frame #14: 0x0000000102a96810 postgres`PortalRun + 392
    frame #15: 0x0000000102a95a6c postgres`exec_simple_query + 1448
    frame #16: 0x0000000102a9303c postgres`PostgresMain + 1152
    frame #17: 0x0000000102a8f57c postgres`BackendMain + 80
    frame #18: 0x0000000102a037ec postgres`postmaster_child_launch + 228
    frame #19: 0x0000000102a078f0 postgres`ServerLoop + 6784
    frame #20: 0x0000000102a05314 postgres`PostmasterMain + 3436
    frame #21: 0x0000000102933efc postgres`main + 816
    frame #22: 0x000000019e4c1d54 dyld`start + 7184

meanwhile duckdb workers looks as follows:

(lldb) th li
Process 23090 stopped
* thread #1: tid = 0x106b496, 0x000000019e6a9c8c libsystem_malloc.dylib`_xzm_free + 1352, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  thread #2: tid = 0x106b497, 0x000000019e846bb0 libsystem_kernel.dylib`semaphore_wait_trap + 8
  thread #3: tid = 0x106b498, 0x000000019e846bb0 libsystem_kernel.dylib`semaphore_wait_trap + 8
  thread #4: tid = 0x106b499, 0x000000019e846bb0 libsystem_kernel.dylib`semaphore_wait_trap + 8
  thread #5: tid = 0x106b49a, 0x000000019e84cf30 libsystem_kernel.dylib`kevent + 8
  thread #6: tid = 0x106b49b, 0x000000019e846bb0 libsystem_kernel.dylib`semaphore_wait_trap + 8
  thread #7: tid = 0x106b49c, 0x000000019e846bb0 libsystem_kernel.dylib`semaphore_wait_trap + 8
  thread #8: tid = 0x106b49d, 0x000000019e846bb0 libsystem_kernel.dylib`semaphore_wait_trap + 8
  thread #9: tid = 0x106b49e, 0x000000019e846bb0 libsystem_kernel.dylib`semaphore_wait_trap + 8
  thread #10: tid = 0x106b49f, 0x000000019e846bb0 libsystem_kernel.dylib`semaphore_wait_trap + 8
  thread #11: tid = 0x106b4a0, 0x000000019e846bb0 libsystem_kernel.dylib`semaphore_wait_trap + 8
  thread #12: tid = 0x106b4a1, 0x000000019e846bb0 libsystem_kernel.dylib`semaphore_wait_trap + 8
(lldb) t 5
* thread #5
    frame #0: 0x000000019e84cf30 libsystem_kernel.dylib`kevent + 8
libsystem_kernel.dylib`kevent:
->  0x19e84cf30 <+8>:  b.lo   0x19e84cf50    ; <+40>
    0x19e84cf34 <+12>: pacibsp
    0x19e84cf38 <+16>: stp    x29, x30, [sp, #-0x10]!
    0x19e84cf3c <+20>: mov    x29, sp
(lldb) bt
* thread #5
  * frame #0: 0x000000019e84cf30 libsystem_kernel.dylib`kevent + 8
    frame #1: 0x0000000102a6f22c postgres`WaitEventSetWait + 408
    frame #2: 0x0000000102a6efe0 postgres`WaitLatch + 96
    frame #3: 0x00000001027e0234 postgres`WaitForParallelWorkersToFinish + 72
    frame #4: 0x00000001028f00d0 postgres`ExecParallelFinish + 164
    frame #5: 0x000000010318a278 pg_duckdb.dylib`pgduckdb::PostgresTableReader::CleanupUnsafe() + 44
    frame #6: 0x000000010318a068 pg_duckdb.dylib`void pgduckdb::__PostgresMemberGuard__<pgduckdb::PostgresTableReader, void>(void (pgduckdb::PostgresTableReader::*)(), pgduckdb::PostgresTableReader*, char const*) + 160
    frame #7: 0x0000000103189f90 pg_duckdb.dylib`pgduckdb::PostgresTableReader::Cleanup() + 68
    frame #8: 0x0000000103185ce0 pg_duckdb.dylib`pgduckdb::PostgresScanGlobalState::UnregisterLocalState() + 80
    frame #9: 0x00000001031867c0 pg_duckdb.dylib`pgduckdb::PostgresScanTableFunction::PostgresScanFunction(duckdb::ClientContext&, duckdb::TableFunctionInput&, duckdb::DataChunk&) + 568
    frame #10: 0x000000010bce3e58 libduckdb.dylib`duckdb::PhysicalTableScan::GetData(duckdb::ExecutionContext&, duckdb::DataChunk&, duckdb::OperatorSourceInput&) const + 80
    frame #11: 0x000000010bea9808 libduckdb.dylib`duckdb::PipelineExecutor::FetchFromSource(duckdb::DataChunk&) + 152
    frame #12: 0x000000010bea5dd0 libduckdb.dylib`duckdb::PipelineExecutor::Execute(unsigned long long) + 260
    frame #13: 0x000000010bea5b1c libduckdb.dylib`duckdb::PipelineTask::ExecuteTask(duckdb::TaskExecutionMode) + 328
    frame #14: 0x000000010be9e1dc libduckdb.dylib`duckdb::ExecutorTask::Execute(duckdb::TaskExecutionMode) + 288
    frame #15: 0x000000010bead19c libduckdb.dylib`duckdb::TaskScheduler::ExecuteForever(std::__1::atomic<bool>*) + 612
    frame #16: 0x000000010beb5318 libduckdb.dylib`void* std::__1::__thread_proxy[abi:ne200100]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void (*)(duckdb::TaskScheduler*, std::__1::atomic<bool>*), duckdb::TaskScheduler*, std::__1::atomic<bool>*>>(void*) + 56
    frame #17: 0x000000019e889c08 libsystem_pthread.dylib`_pthread_start + 136

To Reproduce

make installcheck

OS:

macOS 26.2

pg_duckdb Version (if built from source use commit hash):

6c43a1a

Postgres Version (if built from source use commit hash):

18.1

Hardware:

No response

Full Name:

qsliu

Affiliation:

zbyte inc.

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have tested with a source build

Did you include all relevant data sets for reproducing the issue?

Not applicable - the reproduction does not require a data set

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

• Yes, I have

[qsliu2017]

I try to build with -fsanitize=thread, got the following error.

WARNING: ThreadSanitizer: data race (pid=70861)
  Read of size 8 at 0x00010ad00f98 by thread T11:
    #0 duckdb::TemplatedUniqueIf<pgduckdb::PostgresScanGlobalState, true>::templated_unique_single_t duckdb::make_uniq<pgduckdb::PostgresScanGlobalState, SnapshotData*&, RelationData*&,
duckdb::TableFunctionInitInput&>(SnapshotData*&, RelationData*&, duckdb::TableFunctionInitInput&) <null>:38489732 (pg_duckdb.dylib:arm64+0x11ed28)
    #1 pgduckdb::PostgresScanTableFunction::PostgresScanInitGlobal(duckdb::ClientContext&, duckdb::TableFunctionInitInput&) <null>:38489732 (pg_duckdb.dylib:arm64+0x11e510)
    #2 duckdb::TableScanGlobalSourceState::TableScanGlobalSourceState(duckdb::ClientContext&, duckdb::PhysicalTableScan const&) <null>:38489732 (libduckdb.dylib:arm64+0xe1257c)

  Previous write of size 8 at 0x00010ad00f98 by main thread (mutexes: write M0):
    #0 pgduckdb::PostgresScanFunctionData::PostgresScanFunctionData(RelationData*, unsigned long long, SnapshotData*) <null>:38489732 (pg_duckdb.dylib:arm64+0x11d994)
    #1 pgduckdb::PostgresScanFunctionData::PostgresScanFunctionData(RelationData*, unsigned long long, SnapshotData*) <null>:38489732 (pg_duckdb.dylib:arm64+0x11da90)
    #2 duckdb::TemplatedUniqueIf<pgduckdb::PostgresScanFunctionData, true>::templated_unique_single_t duckdb::make_uniq<pgduckdb::PostgresScanFunctionData, RelationData*&, double&,
SnapshotData*&>(RelationData*&, double&, SnapshotData*&) <null>:38489732 (pg_duckdb.dylib:arm64+0xb9adc)
    #3 pgduckdb::PostgresTable::GetScanFunction(duckdb::ClientContext&, duckdb::unique_ptr<duckdb::FunctionData, std::__1::default_delete<duckdb::FunctionData>, true>&) <null>:38489732
(pg_duckdb.dylib:arm64+0xb996c)
    #4 duckdb::Binder::Bind(duckdb::BaseTableRef&) <null>:38489732 (libduckdb.dylib:arm64+0x155c80)
    #5 Duckdb_ExecCustomScan_Cpp(CustomScanState*) <null>:38489732 (pg_duckdb.dylib:arm64+0x52bbc)
    #6 std::__1::invoke_result<TupleTableSlot* (*)(CustomScanState*), CustomScanState*&>::type pgduckdb::__CPPFunctionGuard__<TupleTableSlot* (*)(CustomScanState*),
&Duckdb_ExecCustomScan_Cpp(CustomScanState*), CustomScanState*>(char const*, char const*, int, CustomScanState*&) <null>:38489732 (pg_duckdb.dylib:arm64+0x52678)
    #7 Duckdb_ExecCustomScan(CustomScanState*) <null>:38489732 (pg_duckdb.dylib:arm64+0x4c9e4)
    #8 ExecCustomScan <null>:38489732 (postgres:arm64+0x100582594)
    #9 ExecProcNodeFirst <null>:38489732 (postgres:arm64+0x10054d66c)
    #10 ExecProcNode <null>:38489732 (postgres:arm64+0x10053eedc)
    #11 ExecutePlan <null>:38489732 (postgres:arm64+0x100537ca4)
    #12 standard_ExecutorRun <null>:38489732 (postgres:arm64+0x100537998)
    #13 ExecutorRun <null>:38489732 (postgres:arm64+0x100537764)
    #14 PortalRunSelect <null>:38489732 (postgres:arm64+0x100a30108)
    #15 PortalRun <null>:38489732 (postgres:arm64+0x100a2f870)
    #16 exec_simple_query <null>:38489732 (postgres:arm64+0x100a28514)
    #17 PostgresMain <null>:38489732 (postgres:arm64+0x100a271f8)
    #18 BackendMain <null>:38489732 (postgres:arm64+0x100a1bc08)
    #19 postmaster_child_launch <null>:38489732 (postgres:arm64+0x100884bec)
    #20 BackendStartup <null>:38489732 (postgres:arm64+0x10088f704)
    #21 ServerLoop <null>:38489732 (postgres:arm64+0x10088b014)
    #22 PostmasterMain <null>:38489732 (postgres:arm64+0x1008899a8)
    #23 main <null>:38489732 (postgres:arm64+0x10062cb44)

  Location is heap block of size 80 at 0x00010ad00f50 allocated by main thread:
    #0 operator new(unsigned long) <null>:38489732 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x84420)
    #1 duckdb::TemplatedUniqueIf<pgduckdb::PostgresScanFunctionData, true>::templated_unique_single_t duckdb::make_uniq<pgduckdb::PostgresScanFunctionData, RelationData*&, double&,
SnapshotData*&>(RelationData*&, double&, SnapshotData*&) <null>:38489732 (pg_duckdb.dylib:arm64+0xb9a7c)
    #2 pgduckdb::PostgresTable::GetScanFunction(duckdb::ClientContext&, duckdb::unique_ptr<duckdb::FunctionData, std::__1::default_delete<duckdb::FunctionData>, true>&) <null>:38489732
(pg_duckdb.dylib:arm64+0xb996c)
    #3 duckdb::Binder::Bind(duckdb::BaseTableRef&) <null>:38489732 (libduckdb.dylib:arm64+0x155c80)
    #4 Duckdb_ExecCustomScan_Cpp(CustomScanState*) <null>:38489732 (pg_duckdb.dylib:arm64+0x52bbc)
    #5 std::__1::invoke_result<TupleTableSlot* (*)(CustomScanState*), CustomScanState*&>::type pgduckdb::__CPPFunctionGuard__<TupleTableSlot* (*)(CustomScanState*),
&Duckdb_ExecCustomScan_Cpp(CustomScanState*), CustomScanState*>(char const*, char const*, int, CustomScanState*&) <null>:38489732 (pg_duckdb.dylib:arm64+0x52678)
    #6 Duckdb_ExecCustomScan(CustomScanState*) <null>:38489732 (pg_duckdb.dylib:arm64+0x4c9e4)
    #7 ExecCustomScan <null>:38489732 (postgres:arm64+0x100582594)
    #8 ExecProcNodeFirst <null>:38489732 (postgres:arm64+0x10054d66c)
    #9 ExecProcNode <null>:38489732 (postgres:arm64+0x10053eedc)
    #10 ExecutePlan <null>:38489732 (postgres:arm64+0x100537ca4)
    #11 standard_ExecutorRun <null>:38489732 (postgres:arm64+0x100537998)
    #12 ExecutorRun <null>:38489732 (postgres:arm64+0x100537764)
    #13 PortalRunSelect <null>:38489732 (postgres:arm64+0x100a30108)
    #14 PortalRun <null>:38489732 (postgres:arm64+0x100a2f870)
    #15 exec_simple_query <null>:38489732 (postgres:arm64+0x100a28514)
    #16 PostgresMain <null>:38489732 (postgres:arm64+0x100a271f8)
    #17 BackendMain <null>:38489732 (postgres:arm64+0x100a1bc08)
    #18 postmaster_child_launch <null>:38489732 (postgres:arm64+0x100884bec)
    #19 BackendStartup <null>:38489732 (postgres:arm64+0x10088f704)
    #20 ServerLoop <null>:38489732 (postgres:arm64+0x10088b014)
    #21 PostmasterMain <null>:38489732 (postgres:arm64+0x1008899a8)
    #22 main <null>:38489732 (postgres:arm64+0x10062cb44)

  Mutex M0 (0x00010a9146b0) created at:
    #0 pthread_mutex_lock <null>:38489732 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x3cf8c)
    #1 std::__1::mutex::lock() <null>:38489732 (libc++.1.dylib:arm64e+0x23884)
    #2 pgduckdb::DuckDBManager::Initialize() <null>:38489732 (pg_duckdb.dylib:arm64+0x30524)
    #3 pgduckdb::DuckDBManager::Get() <null>:38489732 (pg_duckdb.dylib:arm64+0x5db4)
    #4 pgduckdb::DuckDBManager::GetConnection(bool) <null>:38489732 (pg_duckdb.dylib:arm64+0x333e4)
    #5 DuckdbPrepare(Query const*, char const*) <null>:38489732 (pg_duckdb.dylib:arm64+0x60cb8)
    #6 CreatePlan(Query*, bool) <null>:38489732 (pg_duckdb.dylib:arm64+0x61f54)
    #7 std::__1::invoke_result<Plan* (*)(Query*, bool), Query*&, bool&>::type pgduckdb::__CPPFunctionGuard__<Plan* (*)(Query*, bool), &CreatePlan(Query*, bool), Query*, bool>(char const*,
char const*, int, Query*&, bool&) <null>:38489732 (pg_duckdb.dylib:arm64+0x617d0)
    #8 DuckdbPlanNode(Query*, int, bool) <null>:38489732 (pg_duckdb.dylib:arm64+0x60e74)
    #9 DuckdbPlannerHook_Cpp(Query*, char const*, int, ParamListInfoData*) <null>:38489732 (pg_duckdb.dylib:arm64+0x47f5c)
    #10 std::__1::invoke_result<PlannedStmt* (*)(Query*, char const*, int, ParamListInfoData*), Query*&, char const*&, int&, ParamListInfoData*&>::type
pgduckdb::__CPPFunctionGuard__<PlannedStmt* (*)(Query*, char const*, int, ParamListInfoData*), &DuckdbPlannerHook_Cpp(Query*, char const*, int, ParamListInfoData*), Query*, char const*, int,
 ParamListInfoData*>(char const*, char const*, int, Query*&, char const*&, int&, ParamListInfoData*&) <null>:38489732 (pg_duckdb.dylib:arm64+0x47a14)
    #11 DuckdbPlannerHook(Query*, char const*, int, ParamListInfoData*) <null>:38489732 (pg_duckdb.dylib:arm64+0x47150)
    #12 planner <null>:38489732 (postgres:arm64+0x1007c3b68)
    #13 pg_plan_query <null>:38489732 (postgres:arm64+0x100a232e8)
    #14 pg_plan_queries <null>:38489732 (postgres:arm64+0x100a2364c)
    #15 exec_simple_query <null>:38489732 (postgres:arm64+0x100a282f4)
    #16 PostgresMain <null>:38489732 (postgres:arm64+0x100a271f8)
    #17 BackendMain <null>:38489732 (postgres:arm64+0x100a1bc08)
    #18 postmaster_child_launch <null>:38489732 (postgres:arm64+0x100884bec)
    #19 BackendStartup <null>:38489732 (postgres:arm64+0x10088f704)
    #20 ServerLoop <null>:38489732 (postgres:arm64+0x10088b014)
    #21 PostmasterMain <null>:38489732 (postgres:arm64+0x1008899a8)
    #22 main <null>:38489732 (postgres:arm64+0x10062cb44)

  Thread T11 (tid=18208972, running) created by main thread at:
    #0 pthread_create <null>:38489732 (libclang_rt.tsan_osx_dynamic.dylib:arm64e+0x3062c)
    #1 std::__1::thread::thread<void (&)(duckdb::TaskScheduler*, std::__1::atomic<bool>*), duckdb::TaskScheduler*, std::__1::atomic<bool>*, 0>(void (&)(duckdb::TaskScheduler*,
std::__1::atomic<bool>*), duckdb::TaskScheduler*&&, std::__1::atomic<bool>*&&) <null>:38489732 (libduckdb.dylib:arm64+0xfe0bc0)
    #2 pgduckdb::DuckDBManager::Get() <null>:38489732 (pg_duckdb.dylib:arm64+0x5db4)
    #3 pgduckdb::DuckDBManager::GetConnection(bool) <null>:38489732 (pg_duckdb.dylib:arm64+0x333e4)
    #4 DuckdbPrepare(Query const*, char const*) <null>:38489732 (pg_duckdb.dylib:arm64+0x60cb8)
    #5 CreatePlan(Query*, bool) <null>:38489732 (pg_duckdb.dylib:arm64+0x61f54)
    #6 std::__1::invoke_result<Plan* (*)(Query*, bool), Query*&, bool&>::type pgduckdb::__CPPFunctionGuard__<Plan* (*)(Query*, bool), &CreatePlan(Query*, bool), Query*, bool>(char const*,
char const*, int, Query*&, bool&) <null>:38489732 (pg_duckdb.dylib:arm64+0x617d0)
    #7 DuckdbPlanNode(Query*, int, bool) <null>:38489732 (pg_duckdb.dylib:arm64+0x60e74)
    #8 DuckdbPlannerHook_Cpp(Query*, char const*, int, ParamListInfoData*) <null>:38489732 (pg_duckdb.dylib:arm64+0x47f5c)
    #9 std::__1::invoke_result<PlannedStmt* (*)(Query*, char const*, int, ParamListInfoData*), Query*&, char const*&, int&, ParamListInfoData*&>::type
pgduckdb::__CPPFunctionGuard__<PlannedStmt* (*)(Query*, char const*, int, ParamListInfoData*), &DuckdbPlannerHook_Cpp(Query*, char const*, int, ParamListInfoData*), Query*, char const*, int,
 ParamListInfoData*>(char const*, char const*, int, Query*&, char const*&, int&, ParamListInfoData*&) <null>:38489732 (pg_duckdb.dylib:arm64+0x47a14)
    #10 DuckdbPlannerHook(Query*, char const*, int, ParamListInfoData*) <null>:38489732 (pg_duckdb.dylib:arm64+0x47150)
    #11 planner <null>:38489732 (postgres:arm64+0x1007c3b68)
    #12 pg_plan_query <null>:38489732 (postgres:arm64+0x100a232e8)
    #13 pg_plan_queries <null>:38489732 (postgres:arm64+0x100a2364c)
    #14 exec_simple_query <null>:38489732 (postgres:arm64+0x100a282f4)
    #15 PostgresMain <null>:38489732 (postgres:arm64+0x100a271f8)
    #16 BackendMain <null>:38489732 (postgres:arm64+0x100a1bc08)
    #17 postmaster_child_launch <null>:38489732 (postgres:arm64+0x100884bec)
    #18 BackendStartup <null>:38489732 (postgres:arm64+0x10088f704)
    #19 ServerLoop <null>:38489732 (postgres:arm64+0x10088b014)
    #20 PostmasterMain <null>:38489732 (postgres:arm64+0x1008899a8)
    #21 main <null>:38489732 (postgres:arm64+0x10062cb44)

SUMMARY: ThreadSanitizer: data race (pg_duckdb.dylib:arm64+0x11ed28) in duckdb::TemplatedUniqueIf<pgduckdb::PostgresScanGlobalState, true>::templated_unique_single_t
duckdb::make_uniq<pgduckdb::PostgresScanGlobalState, SnapshotData*&, RelationData*&, duckdb::TableFunctionInitInput&>(SnapshotData*&, RelationData*&, duckdb::TableFunctionInitInput&)+0x64