distsql: cancel remote flows on quiesce

We recently saw a test timeout where the cluster shutdown was blocked on
the outbox goroutine attempting to set up a connection to one of the
nodes. We blocked there indefinitely since the remote flows previously
didn't respect quiesce signal (we're using the background context there
because the remote flows outlive the SetupFlow RPC handler context).
This is now fixed.

The bug has been present since forever, and given that we haven't hit
this before, it seems like an extreme edge case, so I decided to omit
the release note.

Release note: None

Author: yuzefovich

pkg/sql/distsql/server.go

@@ -620,6 +620,10 @@ func (ds *ServerImpl) SetupFlow(
 	// Note: the passed context will be canceled when this RPC completes, so we
 	// can't associate it with the flow since it outlives the RPC.
 	ctx = ds.AnnotateCtx(context.Background())
+	// Ensure that the flow respects the node being shut down. Note that since
+	// the flow outlives the RPC, we cannot defer the cancel function, so we
+	// simply ignore it.
+	ctx, _ = ds.Stopper.WithCancelOnQuiesce(ctx)
 	if err := func() error {
 		// Reserve some memory for this remote flow which is a poor man's
 		// admission control based on the RAM usage.