Investigate fine-tuning Spanner client #612
Description
Description
Tessera allows the caller to provide a Spanner client instead of initializing one with the Spanner client library defaults. We had an issue tracking configured clients previously (#428) but didn't find much value in this beyond configuring user agents.
I'd like to revisit this for Spanner, exploring a fine-tuned client that configures session pool values as part of a client config. We've seen issues with DB connections from Trillian to CloudSQL under load, and given we'd like to support a higher max QPS with Rekor v2, we should experiment with different configurations.
Spanner sessions are long-lived connections to the database. By default, the client allows somewhere between 100 and 400 sessions. Each session can either execute a single transaction or multiple when using multiplexed transactions. Tessera load testing has demonstrated that we should be able to sustain > 3000 QPS, so either the Spanner client is already multiplexing transactions or transactions execute fast enough that there's no contention for acquiring a session.
Increasing the number of max sessions and therefore the amount of allowed concurrency will increase DB resource usage, so we'll need to be cautious and keep an eye on metrics while testing.