sftp output does not retry after a failed transfer operation. (Connection failed)

[brokenjacobs]

Version 4.61.0:

It seems that when there is an issue with the sftp, say a permissions issue on overwriting a file, the sftp output will never reconnect to the sftp server. First there is a permission denied issue overwriting a file on the remote:

time="2025-08-12T17:45:56Z" level=error msg="Failed to send message to sftp: failed to open remote file: permission denied" @service=redpanda-connect label="" path=root.output.broker.outputs.0 stream=sftp

Then after the failure I start seeing these messages in the log:

time="2025-08-12T17:45:57Z" level=error msg="Failed to send message to sftp: failed to create SFTP client: ssh: rejected: connect failed (open failed)" @service=redpanda-connect label="" path=root.output.broker.outputs.0 stream=sftp

And on and on forever until this morning:

time="2025-08-13T16:21:14Z" level=error msg="Failed to send message to sftp: failed to create SFTP client: ssh: rejected: connect failed (open failed)" @service=redpanda-connect label="" path=root.output.broker.outputs.0 stream=sftp

If I restart my redpanda-connect pod, transfers resume normally. I believe this is a bug with the reuse of the ssh connection in the sftp output.

[brokenjacobs]

Also, I'm not able to reproduce the original permission denied issue using openssh sftp command line... I'm not sure what is causing that.

It looks like it tries 10 times to write the files and then starts failing with the failed to open messages forever.

[brokenjacobs]

I've tried to figure out how to debug sftp commands using log settings but debug log doesn't appear to log anything from the SFTP library.

[brokenjacobs]

I was able to get logs from the remote server, and it looks like what is happening is too many channels are being opened.

2025-08-13T13:18:30.444982-07:00 user sshd[3833753]: error: no more sessions

https://serverfault.com/questions/182847/count-ssh-sessions

It looks like the channel is not being closed when a permissions issue is encountered, and the maximum number of channels is reached.

I have broker output running 8 ssh connections. The default max sessions is 10, I see 80 retries in the logs which matches perfectly. The session isn't being closed when an error is encountered, nor is it being re-used.