Broken pipe errors and database reconnection

[rudigiesler]

We're seeing an issue with reconnecting to a database. This happens on all of RapidPro's golang components, when a database goes away and comes back again.

When that happens, whenever an inbound message appears, we get an error like the following:

level=error msg=\"error looking up contact\" error=\"write tcp 10.11.6.11:49164-\u003e10.209.0.2:5432: write: broken pipe\"

Doing a bit of digging, seems like this is an issue in sqlx, which is caused by an issue in lib/pq.

The issue in lib/pq seems to be fixed, but it hasn't been merged into sqlx yet, as seen in this PR: jmoiron/sqlx#715

Opening this issue here so that everyone is aware of it if they run into the same issue, and to track that PR and update sqlx when a release with a fix is made.

[rowanseymour]

Hmm sqlx looking a bit unmaintained these days... jmoiron/sqlx#793

Wonder if there's a fork we should be looking at.

[rudigiesler]

According to jmoiron/sqlx#715 (comment) , you can specify the lib/pq version separately from the sqlx version.

I see from v7.1.7, that has been done: 4ade345 . Does that mean that this issue is resolved for those versions onwards?

[rowanseymour]

I don't think we've seen this issue so hard to say but we can keep lib/pq up to date