Description
go version go1.23.3 linux/amd64
I'm given to understand from reading the FoundationDB documentation[1][2] that the key to getting acceptable write throughput is to write through many connections to the database in parallel.
However: when I try to do this, I saw increasing throughput going from 1 to 2, and from 2 to 4 connections;
but I get consistent panics of the "CGO received a signal" type when I try with 8 database connections.
See the crash logs linked below and the repo's kvfile/ directory for a reproducer (run the tests).
https://github.com/glycerine/fdb_demo_golang
https://github.com/glycerine/fdb_demo_golang/blob/master/crash.log.macos.txt
https://github.com/glycerine/fdb_demo_golang/blob/master/crash.log.txt
https://github.com/glycerine/fdb_demo_golang/blob/master/crash.log2.txt
run go test -v in the ./kvfile directory of the fdb_demo_golang repo to reproduce the panic.
The Test002 test in particular; you can change the database pool size here: https://github.com/glycerine/fdb_demo_golang/blob/master/kvfile/db_test.go#L97
Thanks,
Jason
@johscheuer because other Go issues here mention you (but apologies and please re-direct if this is not your area now)
references
[1] https://apple.github.io/foundationdb/features.html#concurrent-connections
"FoundationDB is able to handle large numbers of concurrent client connections. Because it uses a
threadless communications and concurrency model, FoundationDB does not have to create a
thread per connection. This allows full performance even with hundreds of thousands of in-flight
requests."
[2] https://apple.github.io/foundationdb/performance.html
"Its asynchronous design allows it to handle very high concurrency, and for a typical workload with
90% reads and 10% writes, maximum throughput is reached at about 200 concurrent operations.
This number of operations was achieved with 20 concurrent transactions per FoundationDB process
each running 10 operations with 16 byte keys and values between 8 and 100 bytes."
"The implication of this [Little's Law] relation is that, at a given latency, we can maximize throughput
only by concurrently submitting enough outstanding requests. A FoundationDB cluster might have a
commit latency of 2 ms and yet be capable of far more than 500 commits per second. In fact, tens of
thousands of commits per second are easily achievable. To achieve this rate, there must be hundreds
of requests happening concurrently. Not having enough pending requests is the single biggest
reason for low performance."