Opening connections to pgcat is very slow

[JelteF]

Describe the bug I'm running the following command on pgcat.minimal.toml it takes 5 seconds before any queries are being sent by pgbench, indicating that it takes 5 seconds to open 100 connections to localhost. This scales linearly and it takes 50 seconds to open 1000 connections.

❯ pgbench -P 1  -r -h 127.0.0.1 -T 100 --select-only -p 6434 --protocol simple -c 100
pgbench (15.3, server 12.16 (Ubuntu 12.16-1.pgdg22.04+1))
starting vacuum...end.
progress: 5.0 s, 0.0 tps, lat 0.000 ms stddev 0.000, 0 failed
progress: 6.0 s, 21041.9 tps, lat 4.708 ms stddev 1.672, 0 failed

[levkk]

I think we are victims of TCP_NODELAY (https://www.extrahop.com/company/blog/2016/tcp-nodelay-nagle-quickack-best-practices/). Disabling it should improve this for localhost tests especially where the buffering the socket is actually harmful.

[JelteF]

FYI pgbouncer uses TCP_NODELAY. I'm not sure if you meant disabling NOdelay or disabling THE delay.

https://github.com/pgbouncer/pgbouncer/blob/e6ce619785c93392794976c1e936e8c9f589a5ad/src/util.c#L229

[levkk]

Disable the delay. Enable TCP_NODELAY.

[GEverding]

I'm happy to give this a test on my workload.

[zsand]

We've encountered similar performance issues with SELECT queries returning large results. Applying the fix from this pull request https://github.com/postgresml/pgcat/pull/749 significantly improved our performance.