SFTP transfer performance release. Streams uploads/downloads in 255 KiB chunks instead of buffering whole files, pipelines up to 64 concurrent SFTP requests, and raises the server-side MAX_READ_SIZE to the 255 KiB SFTP standard. On a 1 GiB transfer over loopback this drops upload peak RSS from ~3.23 GB to ~20 MB and wall time from 38.6 s to 3.5 s; multi-GB transfers no longer OOM the client.
New Features
None.
Improvements
-
Stream SFTP uploads/downloads instead of buffering whole files in memory (#195).
upload_file/upload_dir_recursivepreviously loaded the entire local file into aVec<u8>viatokio::fs::readbefore callingwrite_all, anddownload_file/download_dir_recursivecalledread_to_endinto a pooled buffer plus aclone()to a separateVecbefore writing locally — so multi-GB transfers had peak RSS that scaled with file size and large files OOM'd the client. Each path now uses astream_copy()helper looping on 255 KiB reads/writes through the existingAsyncRead/AsyncWriteimpls ontokio::fs::Fileandrussh_sftp::client::fs::File. Buffer size matchesMAX_WRITE_LENGTHso each chunk maps to one SFTP packet without further fragmentation.Verified locally on macOS arm64 against
bssh-serverv2.1.3 over loopback with a 1 GiB file:Op Build real RSS upload unpatched 38.65s 3.23 GB upload streaming 3.47s 20 MB download unpatched 3.93s 2.17 GB download streaming 3.41s 16 MB -
Pipeline up to 64 concurrent SFTP requests for upload and download (#196). Bounded pipelined SFTP upload/download helpers replace the previous strictly-sequential request/response loop. Review follow-ups also: cap server-advertised read/write lengths against local maxima to avoid oversized allocations from untrusted SFTP metadata, bound the download reorder queue across both in-flight and pending out-of-order responses, use
fstatsize info where available to avoid reads past EOF and validate unexpected short reads, and preserve remote download handle shutdown after syncing with main. -
Raise bssh-server SFTP
MAX_READ_SIZEfrom 64 KiB to the 255 KiB SFTP standard (#197). The server previously hard-capped every SFTPREADreply at 64 KiB regardless of what the client requested, whilebssh-russh-sftpand OpenSSH'ssftp-serverboth useMAX_READ_LENGTH = 261120(255 KiB). A client asking for a 256 KiB chunk only ever got 64 KiB back, forcing four extra requests per byte stream. Bumped to 261120 so server replies match the chunk size used by the rest of the stack — combined with client-side pipelining (#196), this cuts the per-MiB request count on downloads from 16 to 4. Memory exposure stays bounded: handles are still capped atMAX_HANDLES = 1000per session and each in-flight read still uses a single per-request buffer of this size.
Bug Fixes
None.
CI/CD Improvements
None.
Technical Details
- Streaming buffer size (255 KiB) is aligned with the
russh-sftppacket limit so each chunk stays within one SFTP read/write request without further fragmentation. - Downloaded remote handles are now explicitly closed after successful copies; chunk-size capping and in-memory pipelined upload/download behavior are covered by new SFTP crate tests.
Dependencies
None — Cargo.lock only changed because the workspace package bssh itself bumped from 2.1.3 to 2.1.4.
Breaking Changes
None.
Known Issues
None at release time. The 64-way SFTP pipelining is a fixed concurrency cap (not user-configurable in this release); workloads that need a different parallelism point should track follow-up issues on the repository.
What's Changed
- perf: stream SFTP uploads/downloads instead of buffering whole file by @Yaminyam in #195
- perf: pipeline SFTP requests for upload/download (~2-3x speedup) by @Yaminyam in #196
- perf: raise server MAX_READ_SIZE to SFTP standard 255 KiB by @Yaminyam in #197
Full Changelog: v2.1.3...v2.1.4