Ever Increasing number of Go routines with Pgx v5.5.5 and CockroachDB

[victor-ferrer-form3]

Describe the bug

Recently we upgraded our software to pgx v5.5.5 and inmediately noticed that the number of Go routines our pods use is ever increasing.
This is a screenshot of our monitoring tool, depicting the go_goroutinesmetric:

The point where it starts to crawl up, matches our update to v5.5.2 to v5.5.5.

If we enable Pprof after the pod has spent some hours running we can see this:

goroutine profile: total 318
[...]
74 @ 0x43e32e 0x4099ad 0x4095b2 0x9529ec 0x471501
#	0x9529eb	github.com/jackc/pgx/v5/pgconn/internal/ctxwatch.(*ContextWatcher).Watch.func1+0x8b	/build/vendor/github.com/jackc/pgx/v5/pgconn/internal/ctxwatch/context_watcher.go:51

To Reproduce
Steps to reproduce the behavior:

If possible, please provide runnable example such as:

package main

import (
	"context"
	"log"
	"os"

	"github.com/jackc/pgx/v5"
)

func main() {
	conn, err := pgx.Connect(context.Background(), os.Getenv("DATABASE_URL"))
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close(context.Background())

	// Your code here...
}

Please run your example with the race detector enabled. For example, go run -race main.go or go test -race.

Expected behavior
We would expect those watches to be cancelled/closed and the Go Routines ended.

Actual behavior
Context watchers seem not to be finished properly.

Version

• Go: $ go version -> 1.22
• Cockroach DB:

cockroach version details:
Build Tag:        v23.1.12
Build Time:       2023/11/09 06:15:38
Distribution:     CCL
Platform:         linux amd64 (x86_64-pc-linux-gnu)
Go Version:       go1.19.13
C Compiler:       gcc 6.5.0
Build Commit ID:  d7e9824b4cd6ebf7a8548156f2a772ae6648257d
Build Type:       release
Enabled Assertions: false
(use 'cockroach version --build-tag' to display only the build tag)

• pgx: 5.5.5

[victor-ferrer-form3]

Note: Reverting to v5.5.2 solves the issue.

[drakkan]

Please provide more details. I don't think this happens for every query/use case. For example, I can't replicate it using pgx in database/sql mode (tested both Postgres and CockroachDB).

Probably a Unwatch call is missing in some edge case in the code changed between 5.5.2 and 5.5.5

[victor-ferrer-form3]

Hello @drakkan!

One thing that we have noticed is that the only service in which we see this behavior is one that uses Batch statements.
We have noticed this commit, introduced as part of the release 5.5.4 that has several changes related to Batches, although I am not sure if this causes the issue.

To try to narrow the problem down, we are going to repeat our tests with pgx v.5.5.3 and let you know of the results.

Update: pgx v.5.5.3 does not have this problem.

[sean-]

@victor-ferrer-form3 : have you tried with pgx v5.5.4, or had any success bisecting the problem?

[victor-ferrer-form3]

Hi @sean-,
Yes v.5.5.4 has the problem too. For the moment our only solution was to downgrade to v.5.5.3 and add exceptions for the security vulnerabilities it has.

[victor-ferrer-form3]

Hello @drakkan, do you have any update or ETA on the fix for this?
Thanks

[drakkan]

Hello @drakkan, do you have any update or ETA on the fix for this? Thanks

I'm not working on this, sorry. I don't use Batch statements and so I'm unable to replicate the issue in my use case

[madiyarrakhman]

in this place blocked?

[rcv911]

Hi, @madiyarrakhman, yeah, a potential goroutine leak at <-cw.unwatchChan

github.com/jackc/pgx/v5@v5.6.0
github.com/jackc/pgx/v5/pgconn/ctxwatch/context_watcher.go:49

[redbaron]

We see this pretty consistently after overloading database. Here is how it usually starts:

• DB becomes slow to respond
• request gorotuinges on app side starts to pile up blocking on pix acquire.
• pgx stats show all active, no idle connections.
• as DB is overloaded connections start to become cancelled on timeout

Then weird thing happens:

• we see increase in stuck coroutines on <-cw.unwatchChan
• sum of constructing + idle + acquired is less than total maximum, despite DB is still being overloaded.

So this event coincides with some connections stopped being active, but not becoming idle

[redbaron]

Confirmed that this bug is present on 5.7.5 and we don't use CockroachDB, just standard PostgreSQL

[hdonnay]

I took a stab at this with #2515, and managed to work up what I think is a reproducer test.