go/adbc/driver/snowflake+flightsql: Memory access violation when both are used in Python

[joellubi]

What happened?

When using both the snowflake driver and the flightsql driver in the same python program, the process crashes with a segmentation violation. Using one of these drivers at the same time as the postgres driver does not cause a crash, which makes it seem like there might be an issue in the common Go memory management of the snowflake and flightsql drivers.

How can we reproduce the bug?

Minimal Reproduction:

import adbc_driver_flightsql.dbapi
import adbc_driver_snowflake.dbapi

adbc_driver_flightsql.dbapi.connect("grpc+tcp://localhost:8080")
adbc_driver_snowflake.dbapi.connect()

Result:

[signal SIGSEGV: segmentation violation code=0x1 addr=0x14 pc=0x1636e3c9d]

goroutine 1 [running, locked to thread]:
runtime.throw({0x165155d38?, 0x1c0000061a0?})
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/panic.go:1077 +0x5c fp=0x1c000731e88 sp=0x1c000731e58 pc=0x163713c5c
runtime.sigpanic()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/signal_unix.go:845 +0x3e9 fp=0x1c000731ee8 sp=0x1c000731e88 pc=0x163729fe9
runtime.(*waitq).dequeue(...)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/chan.go:781
runtime.closechan(0x1c000182000)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/chan.go:380 +0x17d fp=0x1c000731f40 sp=0x1c000731ee8 pc=0x1636e3c9d
runtime.main()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:256 +0x28f fp=0x1c000731fe0 sp=0x1c000731f40 pc=0x16371664f
runtime.goexit()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0x1c000731fe8 sp=0x1c000731fe0 pc=0x1637468c1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:398 +0xce fp=0x1c000076fa8 sp=0x1c000076f88 pc=0x163716ace
runtime.goparkunlock(...)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:404
runtime.forcegchelper()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:322 +0xb3 fp=0x1c000076fe0 sp=0x1c000076fa8 pc=0x163716953
runtime.goexit()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0x1c000076fe8 sp=0x1c000076fe0 pc=0x1637468c1
created by runtime.init.6 in goroutine 1
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:310 +0x1a

goroutine 3 [GC sweep wait]:
runtime.gopark(0x1?, 0x0?, 0x0?, 0x0?, 0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:398 +0xce fp=0x1c000077778 sp=0x1c000077758 pc=0x163716ace
runtime.goparkunlock(...)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:404
runtime.bgsweep(0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgcsweep.go:321 +0xdf fp=0x1c0000777c8 sp=0x1c000077778 pc=0x163702eff
runtime.gcenable.func1()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:200 +0x25 fp=0x1c0000777e0 sp=0x1c0000777c8 pc=0x1636f8025
runtime.goexit()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0x1c0000777e8 sp=0x1c0000777e0 pc=0x1637468c1
created by runtime.gcenable in goroutine 1
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:200 +0x66

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x1c00007e000?, 0x1653aa1b8?, 0x0?, 0x0?, 0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:398 +0xce fp=0x1c000077f70 sp=0x1c000077f50 pc=0x163716ace
runtime.goparkunlock(...)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:404
runtime.(*scavengerState).park(0x16698b740)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgcscavenge.go:425 +0x49 fp=0x1c000077fa0 sp=0x1c000077f70 pc=0x163700749
runtime.bgscavenge(0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgcscavenge.go:658 +0x59 fp=0x1c000077fc8 sp=0x1c000077fa0 pc=0x163700cf9
runtime.gcenable.func2()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:201 +0x25 fp=0x1c000077fe0 sp=0x1c000077fc8 pc=0x1636f7fc5
runtime.goexit()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0x1c000077fe8 sp=0x1c000077fe0 pc=0x1637468c1
created by runtime.gcenable in goroutine 1
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:201 +0xa5

goroutine 18 [finalizer wait]:
runtime.gopark(0x198?, 0x1657b6dc0?, 0x1?, 0x7c?, 0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:398 +0xce fp=0x1c000076620 sp=0x1c000076600 pc=0x163716ace
runtime.runfinq()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mfinal.go:193 +0x107 fp=0x1c0000767e0 sp=0x1c000076620 pc=0x1636f7047
runtime.goexit()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0x1c0000767e8 sp=0x1c0000767e0 pc=0x1637468c1
created by runtime.createfing in goroutine 1
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mfinal.go:163 +0x3d

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:398 +0xce fp=0x1c000072750 sp=0x1c000072730 pc=0x163716ace
runtime.gcBgMarkWorker()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1295 +0xe5 fp=0x1c0000727e0 sp=0x1c000072750 pc=0x1636f9ba5
runtime.goexit()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0x1c0000727e8 sp=0x1c0000727e0 pc=0x1637468c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1219 +0x1c

goroutine 20 [GC worker (idle)]:
runtime.gopark(0xaaf807b9a3a23?, 0x0?, 0x0?, 0x0?, 0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:398 +0xce fp=0x1c000072f50 sp=0x1c000072f30 pc=0x163716ace
runtime.gcBgMarkWorker()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1295 +0xe5 fp=0x1c000072fe0 sp=0x1c000072f50 pc=0x1636f9ba5
runtime.goexit()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0x1c000072fe8 sp=0x1c000072fe0 pc=0x1637468c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1219 +0x1c

goroutine 34 [GC worker (idle)]:
runtime.gopark(0xaaf807b9a398a?, 0x3?, 0xde?, 0x37?, 0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:398 +0xce fp=0x1c000508750 sp=0x1c000508730 pc=0x163716ace
runtime.gcBgMarkWorker()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1295 +0xe5 fp=0x1c0005087e0 sp=0x1c000508750 pc=0x1636f9ba5
runtime.goexit()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0x1c0005087e8 sp=0x1c0005087e0 pc=0x1637468c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1219 +0x1c

goroutine 21 [GC worker (idle)]:
runtime.gopark(0xaaf807b9a3a29?, 0x0?, 0x0?, 0x0?, 0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:398 +0xce fp=0x1c000073750 sp=0x1c000073730 pc=0x163716ace
runtime.gcBgMarkWorker()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1295 +0xe5 fp=0x1c0000737e0 sp=0x1c000073750 pc=0x1636f9ba5
runtime.goexit()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0x1c0000737e8 sp=0x1c0000737e0 pc=0x1637468c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1219 +0x1c

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x1669be420?, 0x3?, 0xd0?, 0xb5?, 0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:398 +0xce fp=0x1c000508f50 sp=0x1c000508f30 pc=0x163716ace
runtime.gcBgMarkWorker()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1295 +0xe5 fp=0x1c000508fe0 sp=0x1c000508f50 pc=0x1636f9ba5
runtime.goexit()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0x1c000508fe8 sp=0x1c000508fe0 pc=0x1637468c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1219 +0x1c

goroutine 22 [GC worker (idle)]:
runtime.gopark(0xaaf807b9a36d1?, 0x3?, 0xf3?, 0xff?, 0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:398 +0xce fp=0x1c000073f50 sp=0x1c000073f30 pc=0x163716ace
runtime.gcBgMarkWorker()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1295 +0xe5 fp=0x1c000073fe0 sp=0x1c000073f50 pc=0x1636f9ba5
runtime.goexit()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0x1c000073fe8 sp=0x1c000073fe0 pc=0x1637468c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1219 +0x1c

goroutine 36 [GC worker (idle)]:
runtime.gopark(0xaaf807b99e23f?, 0x1?, 0x87?, 0xd4?, 0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:398 +0xce fp=0x1c000509750 sp=0x1c000509730 pc=0x163716ace
runtime.gcBgMarkWorker()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1295 +0xe5 fp=0x1c0005097e0 sp=0x1c000509750 pc=0x1636f9ba5
runtime.goexit()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0x1c0005097e8 sp=0x1c0005097e0 pc=0x1637468c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1219 +0x1c

goroutine 23 [GC worker (idle)]:
runtime.gopark(0xaaf807baf3078?, 0x0?, 0x0?, 0x0?, 0x0?)
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/proc.go:398 +0xce fp=0x1c000074750 sp=0x1c000074730 pc=0x163716ace
runtime.gcBgMarkWorker()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1295 +0xe5 fp=0x1c0000747e0 sp=0x1c000074750 pc=0x1636f9ba5
runtime.goexit()
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/asm_amd64.s:1650 +0x1 fp=0x1c0000747e8 sp=0x1c0000747e0 pc=0x1637468c1
created by runtime.gcBgMarkStartWorkers in goroutine 1
	/Users/runner/hostedtoolcache/go/1.21.8/x64/src/runtime/mgc.go:1219 +0x1c

Environment/Setup

Reproduces with:

adbc-driver-snowflake = { file = "/path/to/adbc_driver_snowflake-1.0.0-py3-none-macosx_10_15_x86_64.whl" }
adbc-driver-flightsql = { file = "/path/to/adbc_driver_flightsql-1.0.0-py3-none-macosx_10_15_x86_64.whl" }

and:

adbc-driver-snowflake = "^0.11.0"
adbc-driver-flightsql = "^0.11.0"

Does NOT reproduce with:

adbc-driver-snowflake = "^0.10.0"
adbc-driver-flightsql = "^0.10.0"

However, as I've tried pushing against various edge cases I have been able to get some segmentation violations with 0.10.0 too, but haven't yet isolated the specific behavior that causes it in that version.

[zeroshade]

Wow, that's really odd. My first thought to ask here is whether or not this reproduces on a non-Mac platform or if this is specific to MacOS?

[lidavidm]

I don't get this (macOS aarch64) when building from main.

>>> import adbc_driver_flightsql.dbapi
>>> import adbc_driver_snowflake.dbapi
>>>
>>> adbc_driver_flightsql.dbapi.connect("grpc+tcp://localhost:8080")
/Users/lidavidm/Code/arrow-adbc/python/adbc_driver_manager/adbc_driver_manager/dbapi.py:307: Warning: Cannot disable autocommit; conn will not be DB-API 2.0 compliant
  warnings.warn(
<adbc_driver_manager.dbapi.Connection object at 0x13cb4cb50>
>>> adbc_driver_snowflake.dbapi.connect()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/lidavidm/Code/arrow-adbc/python/adbc_driver_snowflake/adbc_driver_snowflake/dbapi.py", line 120, in connect
    conn = adbc_driver_manager.AdbcConnection(db, **(conn_kwargs or {}))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "adbc_driver_manager/_lib.pyx", line 681, in adbc_driver_manager._lib.AdbcConnection.__init__
  File "adbc_driver_manager/_lib.pyx", line 237, in adbc_driver_manager._lib.check_error
adbc_driver_manager.OperationalError: IO: 260000: account is empty

[lidavidm]

It also doesn't reproduce with the 0.11.0 wheels so this appears to be a macOS x64 issue, which is going to be difficult to debug as I only have an aarch64 machine.

Are you able to reproduce it on any other platforms?

[lidavidm]

ok, looks like it does happen under rosetta

[lidavidm]

However, debugging under rosetta is not working great as lldb appears to get stuck/execution appears to be extremely slow.

[lidavidm]

Actually under lldb the process just hangs forever, even if we only import/use a single driver. So that won't help.

[lidavidm]

@zeroshade is there a problem perhaps with having two copies of the go runtime in one process?

[lidavidm]

I do also see that there are public symbols in the driver from cgo et al

[lidavidm]

golang/go#65050

[lidavidm]

I think we're between a rock and a hard place here.

• C/C++ is not suitable for the Flight SQL driver because of libarrow/Protobuf/gRPC.
• C/C++ is not suitable for the Snowflake driver given gosnowflake is much better supported.
• Rust/C# are not quite there yet. (Also @CurtHagenlocher, would .NET have the same issues as Go above with multiple shared libraries and hence multiple runtimes in one process?)

I think the interim solution would be to build a "combined" driver that has both drivers in one library.

I'm hesitant to hold up the release though since this has been a latent issue for a while.

CC @zeroshade @kou @paleolimbot for opinions on the last part too.

[lidavidm]

@cocoa-xu this will also affect BigQuery...it seems both C++ and Go have packaging issues here...

[lidavidm]

FWIW, the issue above also seems to indicate it can only be reproduced on amd64 macOS.

[zeroshade]

Since this is a latent issue for quite some time, and at least up to this point it appears that our consumers don't yet tend to have multiple drivers in one process (or if they do, it's not on macos and isn't exhibiting the issue) I'm okay with this not being a release blocker. We're going to have to figure something out, but I think it's fine to release without a fix for this. Maybe we just add something to the documentation to point out the problem for now and note that we're working on it?

[paleolimbot]

I agree that it is not a release blocker...the improvements to the individual drivers are (in my opinion) important to release whilst we improve the situation for using multiple drivers in one process. I do have an x86 mac locally that I might be use to iterate on some of this...in R we're using the c-static build mode instead of the c-shared one (but I assume we would have the same issue there).