[aiohttp] - disable reset query and remove explicit call prepare

[Reskov]

• asyncpg reset query called each time the connection is released from pool. This do a lot of stuff like close active listeners, release advisory locks, etc. not used in our benchmark so we can omit it.
• asyncpg call prepare statement internally, and cache the result inside connection cache => no need to call it explicitly.

Before

---------------------------------------------------------
 Queries: 1 for query
 wrk -H 'Host: tfb-server' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 30 -c 32 --timeout 8 -t 6 "http://tfb-server:8080/queries/1"
---------------------------------------------------------
Running 30s test @ http://tfb-server:8080/queries/1
  6 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.39ms    1.08ms  48.55ms   91.12%
    Req/Sec     3.87k     0.93k    5.33k    75.06%
  Latency Distribution
     50%    1.13ms
     75%    1.68ms
     90%    2.36ms
     99%    5.08ms
  693193 requests in 30.01s, 123.47MB read
Requests/sec:  23100.21
Transfer/sec:      4.11MB
STARTTIME 1744784708
ENDTIME 1744784738

---------------------------------------------------------
 Queries: 20 for query
 wrk -H 'Host: tfb-server' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 30 -c 32 --timeout 8 -t 6 "http://tfb-server:8080/queries/20"
---------------------------------------------------------
Running 30s test @ http://tfb-server:8080/queries/20
  6 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     7.83ms    5.36ms  92.00ms   84.83%
    Req/Sec   688.64    268.40     1.25k    59.11%
  Latency Distribution
     50%    6.34ms
     75%    9.36ms
     90%   13.78ms
     99%   29.06ms
  123464 requests in 30.03s, 93.19MB read
Requests/sec:   4111.13
Transfer/sec:      3.10MB
STARTTIME 1744784836
ENDTIME 1744784866

After

 Queries: 1 for query
 wrk -H 'Host: tfb-server' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 30 -c 32 --timeout 8 -t 6 "http://tfb-server:8080/queries/1"
---------------------------------------------------------
Running 30s test @ http://tfb-server:8080/queries/1
  6 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   786.25us  642.33us  24.76ms   92.01%
    Req/Sec     6.82k   782.73    16.25k    84.51%
  Latency Distribution
     50%  651.00us
     75%    0.95ms
     90%    1.33ms
     99%    2.79ms
  1222301 requests in 30.10s, 217.72MB read
Requests/sec:  40608.53
Transfer/sec:      7.23MB
STARTTIME 1744785790
ENDTIME 1744785820
---------------------------------------------------------
...

 Queries: 20 for query
 wrk -H 'Host: tfb-server' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 30 -c 32 --timeout 8 -t 6 "http://tfb-server:8080/queries/20"
---------------------------------------------------------
Running 30s test @ http://tfb-server:8080/queries/20
  6 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.68ms    1.43ms  36.10ms   77.92%
    Req/Sec     1.38k   131.73     1.81k    71.94%
  Latency Distribution
     50%    3.47ms
     75%    4.32ms
     90%    5.27ms
     99%    7.91ms
  247057 requests in 30.01s, 186.49MB read
Requests/sec:   8232.68
Transfer/sec:      6.21MB
STARTTIME 1744785918
ENDTIME 1744785949

[Reskov]

@Dreamsorcerer please take a look. Performance almost twice better for /queries endpoint on my local machine.

[Dreamsorcerer]

@Dreamsorcerer please take a look. Performance almost twice better for /queries endpoint on my local machine.

If there's such a big difference, I wonder if something should be changed upstream? Looking through the code, it looks like it already skips that if the server doesn't have those capabilities:
https://github.com/MagicStack/asyncpg/blob/5b14653e0b447d956aa01ec658562138e19f0293/asyncpg/connection.py#L1747-L1754

If it's CrateDB or CockroachDB then it skips all of that anyway, and on Amazon RedShift only does the RESET ALL. Only on standard PostgresQL does it run all of that query:
https://github.com/MagicStack/asyncpg/blob/5b14653e0b447d956aa01ec658562138e19f0293/asyncpg/connection.py#L2661-L2697

Which makes me wonder if it's really necessary after all...

[Dreamsorcerer]

I wonder if something should be changed upstream?

For example, the UNLISTEN is probably not needed if there's no listeners on the connection at reset time:
https://github.com/MagicStack/asyncpg/blob/5b14653e0b447d956aa01ec658562138e19f0293/asyncpg/connection.py#L1529

[Reskov]

I do not think that disable only unlisten will make a noticeable difference. The big difference is to execute or not whole reset query. Some of the options like parameter set for RESET ALL, looks not so easy to be detected.

On the upstream we can probably expose connection ServerCapabilities to make it public and allow end-user to decide what options he want to have.

P.s We can also tune this in other way, since benchmark use keep alive connections, we may try to pin db connections to the specific established connection without acquire/releasing them from pool on each request.

[Dreamsorcerer]

I do not think that disable only unlisten will make a noticeable difference. The big difference is to execute or not whole reset query. Some of the options like parameter set for RESET ALL, looks not so easy to be detected.

If you try hardcoding the query locally, can you verify if that's true? My suspicion is that with the async nature, that maybe the contents of the query is having an impact (there are 4 statementns in there after all). I'm a little surprised to see double the performance for that, as it suggests that the reset query is actually taking more time than the regular queries (given that the DB query can't be 100% of the benchmark time).

[Dreamsorcerer]

If you try hardcoding the query locally

So, hardcode to:

return 'SELECT pg_advisory_unlock_all();CLOSE ALL;UNLISTEN *;RESET ALL;'

Then remove each statement and verify how much impact each part of the query has.

[Reskov]

This is hard to check locally since a have a large deviation due to I run tests on virtual machine and low test duration. So if we want to know exact percentage of improvement. We have to prepare bare-metal machine and test only asyncpg without nginx/aiohttp overhead.

SELECT pg_advisory_unlock_all();CLOSE ALL;UNLISTEN *;RESET ALL;

 Queries: 1 for query
 wrk -H 'Host: tfb-server' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 30 -c 32 --timeout 8 -t 6 "http://tfb-server:8080/queries/1"
---------------------------------------------------------
Running 30s test @ http://tfb-server:8080/queries/1
  6 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.05ms  708.96us  35.15ms   86.89%
    Req/Sec     5.01k     1.03k    6.28k    82.28%
  Latency Distribution
     50%    0.89ms
     75%    1.29ms
     90%    1.79ms
     99%    3.37ms
  896495 requests in 30.00s, 159.69MB read
Requests/sec:  29880.08
Transfer/sec:      5.32MB
STARTTIME 1744814199
ENDTIME 1744814229

Select 1

class NoResetConnection(asyncpg.Connection):
    __slots__ = ()

    def get_reset_query(self):
        return 'SELECT 1'

  Queries: 1 for query
 wrk -H 'Host: tfb-server' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 30 -c 32 --timeout 8 -t 6 "http://tfb-server:8080/queries/1"
---------------------------------------------------------
Running 30s test @ http://tfb-server:8080/queries/1
  6 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.91ms  530.35us  15.76ms   81.01%
    Req/Sec     5.67k   418.48    13.11k    81.34%
  Latency Distribution
     50%  788.00us
     75%    1.13ms
     90%    1.54ms
     99%    2.62ms
  1015678 requests in 30.10s, 180.91MB read
Requests/sec:  33744.19
Transfer/sec:      6.01MB
STARTTIME 1744814374
ENDTIME 1744814405

class NoResetConnection(asyncpg.Connection):
    __slots__ = ()

    def get_reset_query(self):
        return ''

---------------------------------------------------------
 Queries: 1 for query
 wrk -H 'Host: tfb-server' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 30 -c 32 --timeout 8 -t 6 "http://tfb-server:8080/queries/1"
---------------------------------------------------------
Running 30s test @ http://tfb-server:8080/queries/1
  6 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   674.03us  382.59us  10.58ms   78.08%
    Req/Sec     7.64k   403.10    12.09k    76.37%
  Latency Distribution
     50%  588.00us
     75%  842.00us
     90%    1.15ms
     99%    1.87ms
  1370230 requests in 30.10s, 244.07MB read
Requests/sec:  45524.99
Transfer/sec:      8.11MB
STARTTIME 1744814546
ENDTIME 1744814576

[Dreamsorcerer]

Right, so looks like sending the query is most of the overhead, but looks like there could still be atleast a 10% improvement if upstream were to reduce the statements sent.

[Reskov]

I'm a little surprised to see double the performance for that, as it suggests that the reset query is actually taking more time than the regular queries (given that the DB query can't be 100% of the benchmark time).

Yeah, it is actually not double. Initially I have two optimisations inside this PR (remove call of prepare and remove reset query). Let's assume (1-33744.19/40608.53) around 17%