Update docker-compose-common-components.yaml (#1207)

[ElGuillermo]

The default 100 max postgres connections isn't enough when managing more than 4 (?) game servers, triggering http 502 errors in UI (see : https://discord.com/channels/685692524442026020/685695097349734469/1470638305606373452).

This fix increases the default to 300. This arbitrary number should suffice for a number of managed game servers up to 8, but still has to be evaluated in real conditions and edge cases (8+ servers).

[Fragger]

This seems sensible to me, something I've had to change a couple of times as well.

[ElGuillermo]

I'm running some tests on my CRCON, but as it only manages one game server, I can't tell much about it. Only thing I know is : people having the issue got it fixed using 300. Highest I got here was 40 (one game server).

[Fragger]

Quick info from a couple of setups. One with 4 full public servers and an event server is sitting at 112 connections. Another with 10 event servers is sitting at 147 connections but they are not being used at the moment.

[ElGuillermo]

I guess these numbers should be tested using 8 or more full game servers, monitored by several CRCON users doing some heavy load things (searching logs, hammering the views, etc)...

I'm testing an alternative : using pgbouncer to pool the postgre connections so they never hit any hardcoded ceiling. Works quite nicely since days.

[FlorianSW]

Just my two cents: Sure we could increase the limit (and as a quick fix this is probably good to go). However, I would really like to take a deeper look into the actual number of connections we need and, more importantly, why.

That's what I can think of out of my head without looking too deeply into the code:

• First of all the backend threads:
  • by default, we use a single process (NB_API_WORKERS) with 8 threads (NB_API_THREADS)
  • from what I understood (and that might be wrong!) is that each thread will use it's own instance of SQLAlchemy's Engine, which holds the connection pool, among other things
  • As each of the backend threads should work on a single HTTP requests at the same time, there shouldn't be any more than a total of NB_API_WORKERS * NB_API_THREADS (in our default config 8), if there are more, then we're most likely doing something in parallel when answering HTTP requests, which we should reconsider (at least when it comes to DB connections)
• all the supervisor programs:
  • each of the defined programs in supervisord.conf are started as their own process, independant of all the other ones
  • they therefore use their own Engine and connection pool because of that
  • we currently have 16 programs (not all fo them might be started on every setup, but let's suppose they do)
  • each of them will, on the other hand, only open a connection to the DB, if actually needed
  • I presume that a good portion of them actually do not need a DB connection, if there were not the DB based config, a lot of the services load their configuration from the DB on startup. I wasn't able, yet, to find out from SQLAlchemy when a connection is released from the pool if unused, but chances are that they stay in the pool (???) for the entire time the service is running, even though it is only used once during startup?

Together that would make about 24 connections (at minimum) for the above mentioned services. I assume that the number increases for some services that do stuff in parallel (each pool can have up to 5 conections, which would allow a total number of 100 connections per server, which will, hopefully, never ever reached, but as a max ceiling).

Maybe we can take a deeper look into where the total number of 40 connections come from and tackle this issue (as wel)? :)

[ElGuillermo]

Maybe we also can add this to the postgresql.conf :

idle_in_transaction_session_timeout = 60000  # Timeout in milliseconds

[FlorianSW]

Maybe we also can add this to the postgresql.conf :

idle_in_transaction_session_timeout = 60000  # Timeout in milliseconds

None of the connections in the connection pool should have transactions pending, shouldn't they? At least we're always either commiting a transaction or rolling it back. And the config you mentioned, from what I understand, applies to Postgres sessions with open transactions, only 🤔 Hence this will probably not change much.

[FlorianSW]

I opened two PRs:

• [Operating] allow to identify DB connection "owner" #1261 proposes a way to first of all identify what component is responsible for what connection (acting on data is probably the first good step)
• Allow some services to not use a connection pool to the DB #1262 From what I said above: At least 4 services currently are built in a way that does not require them to hold a connection pool (or a DB connection for that matter) at all, hence pooling is disabled with that PR for them

[FlorianSW]

I've created #1263 to move this discussion there and not have this PR thread overrun by it, sorry :]