Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
#### Changed
- **The CLI remote scan of `sync`, `check` and `cryptcheck` runs on the provider's list pool, up to `--checkers` directories at once.** The CLI walked the remote tree with a serial walker of its own while the GUI scan already listed on clone workers; on the lab SFTP server a 5000-file `sync` spent 10.8 s in a no-op run and 14.7 s in a delta run against rclone's 4.2 s and 5.4 s, all of it in the scan. The three commands now use the shared walker (same filters, same completeness accounting, spinner fed by its progress hook), so a provider that lists on independent connections (SFTP, S3, WebDAV, B2, the HTTP clouds) lists in parallel and a single-session provider walks as before. `--checkers` therefore bounds something real now; its help text says so. Also fixed in the shared walker: a symlink to a directory is listed but no longer walked (the CLI walker already refused it, the GUI walker did not).
- **The GUI sync scan lists SFTP directories on up to four connections, and scan workers stay warm across directories.** The parallel scanner cloned a fresh worker for every directory, which is a client copy for HTTP providers but would be a full SSH handshake on SFTP, so SFTP was left on the single locked session (the DAG engine review measured 4 sequential round trips per directory: 15 to 18 s for 5000 files in 51 folders, rclone 4 s). SFTP now exposes the clone-pool scan hooks once a connection exists, and the scanner keeps a worker that listed a directory cleanly for the next directory instead of re-dialling (HTTP clones do not opt in and keep their per-directory clone). The CLI `sync`, `check` and `cryptcheck` run on the same scanner since the entry above.
- **Multi-file SFTP transfers honour `--parallel` up to 16 connections, and the CLI says so when a provider ceiling binds a higher request.** The SFTP pool ceiling was a literal 4, so `--parallel 16` delivered 4 connections without a word. The DAG engine review battery on the Hetzner lab (wired gigabit, 5000 x 4 KiB files) measured what that cost: from 4 to 16 rclone gained 1.9x on upload and 3.6x on download, AeroFTP moved 1%. The ceiling is now 16, the same range as `--sftp-concurrency`; the effective count stays the lower of the ceiling and `--parallel`, so the default of 4 connections is unchanged. On every provider, `put -r` and `get -r` now print a note when the provider ceiling is lower than the requested `--parallel` (text mode; quiet and JSON stay silent).
- **SFTP single-stream downloads read ahead by default (window 32), instead of one 256 KiB read per round trip.** The read-ahead tier existed behind `--sftp-readahead` and `AEROFTP_SFTP_READAHEAD`; the DAG engine review measured what the default cost on a 53 ms link with a 300 MiB file: 118 to 129 s serial, 38.8 s with a window of 16, 34.5 s with 32, 35.6 s with 64, rclone 30 s. The window now defaults to 32 when nothing asks for one; `AEROFTP_SFTP_READAHEAD=off` (or `0`) restores the serial read, an explicit `--sftp-readahead` or preset still wins, a bandwidth cap still takes the serial loop that owns the precise throttle, and the job-wide buffer budget still bounds the window per connection.
- **`put -r --skip-restricted` uploads a tree whose names the destination partly forbids, setting aside and reporting the offending entries instead of refusing the whole batch.** The default is unchanged and deliberate: one restricted name (a tab in a file name bound for S3, for instance) still stops the batch before anything is created. The review battery of 2026-09-05 measured the two policies side by side on the same 20-file tree: AeroFTP uploaded 0 files, rclone 19 and reported the one it could not. With the flag a folder with a restricted name takes its children along, every skip is listed once, and the run ends `partial` with exit 4 because not everything requested landed.
- **Single-file downloads are multi-threaded by default: files of 250 MiB or more are fetched with 4 concurrent range streams, the same stream count as rclone's default (4 streams above 256Mi; AeroFTP's cutoff stays 250 MiB, so files between the two thresholds split here and not there).** `--multi-thread-streams` used to default to `1`, so the DAG range scheduler, the strict 206 probe and the single-stream fallback all existed but only ran when a user found the flag; the May 2026 live matrix showed rclone ahead on every 1 GiB download because of that default alone, while the shared harness measured the two engines within noise once the streams were equal. The gate is unchanged: only backends that prove Range honesty (S3, Azure, SFTP as independent connections, WebDAV and Koofr after a live probe) split the file, everything else and every file below `--multi-thread-cutoff` stays single-stream, and `--multi-thread-streams 1` or `AEROFTP_MULTI_THREAD_STREAMS=1` restores the old behaviour. `pget` is unaffected.
Expand Down
2 changes: 1 addition & 1 deletion docs/CLI-GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -1915,7 +1915,7 @@ It also emits the transfer-scheduler surface: a `protocol_transfer_capabilities`
| `--limit-rate <speed>` | Speed limit (e.g., `1M`, `500K`) |
| `--sftp-readahead <N>` | SFTP read-ahead window (2 to 1024). Default 32 when unset, which keeps up to 32 reads in flight on one connection; `AEROFTP_SFTP_READAHEAD=off` (or `0`) restores one read per round trip; a bandwidth cap always uses the serial loop |
| `--bwlimit <schedule>` | Bandwidth schedule (e.g., `"08:00,512k 18:00,off"` or `"1M"`) |
| `--parallel <n>` | Number of parallel transfer workers for recursive/bulk operations |
| `--parallel <n>` | Number of parallel transfer workers for recursive/bulk operations (default 4, up to 32). Each provider has its own ceiling (SFTP 16 connections, FTP 5, S3 and B2 their clone pools, single-session providers 1); when the ceiling is lower than the request the CLI prints a note in text mode |
| `--checkers <n>` | Directories listed at once by the remote scan of `sync`, `check` and `cryptcheck` (default 8, range 1-64), capped by the provider's list pool; single-session providers walk one directory at a time |
| `--partial` | Resume interrupted transfers when the provider supports partial files or remote offsets |
| `--include <pattern>` | Include only files matching glob pattern (repeatable) |
Expand Down
22 changes: 22 additions & 0 deletions src-tauri/src/bin/aeroftp_cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7993,6 +7993,26 @@ fn create_spinner(msg: &str) -> ProgressBar {
pb
}

/// A flag that accepts 32 and delivers 4 without a word is worse than a lower
/// documented limit: when the provider's session ceiling binds the requested
/// `--parallel`, say so once on stderr (text mode only; quiet and JSON stay
/// silent). The DAG engine review battery found SFTP pinned at 4 for every
/// `--parallel` value while rclone scaled 3.6x from 4 to 16 on the same tree.
fn note_parallel_ceiling(
cli: &Cli,
settings: &ftp_client_gui_lib::transfer_settings::ResolvedTransferSettings,
) {
if cli.quiet || cli.json || cli.machine {
return;
}
if settings.max_concurrent < settings.requested_max_concurrent {
eprintln!(
"Note: this provider caps parallel workers at {} (requested --parallel {})",
settings.max_concurrent, settings.requested_max_concurrent
);
}
}

fn effective_parallel_workers(cli: &Cli) -> usize {
cli.parallel.clamp(1, 32)
}
Expand Down Expand Up @@ -9424,6 +9444,7 @@ async fn run_shared_provider_download_batch(
| ProviderExecutorSessionModel::SftpConnectionPool { .. }
| ProviderExecutorSessionModel::FtpConnectionPool { .. }
);
note_parallel_ceiling(cli, &runtime_settings);
if !is_pool_backed {
// Not pool-backed: return the still-connected provider so the
// caller runs the legacy independent-connection batch.
Expand Down Expand Up @@ -9646,6 +9667,7 @@ async fn run_shared_provider_upload_batch(
| ProviderExecutorSessionModel::SftpConnectionPool { .. }
| ProviderExecutorSessionModel::FtpConnectionPool { .. }
);
note_parallel_ceiling(cli, &runtime_settings);
if !is_pool_backed {
// Not pool-backed: return the still-connected provider so the
// caller runs the legacy independent-connection batch.
Expand Down
41 changes: 37 additions & 4 deletions src-tauri/src/providers/sftp.rs
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ fn configure_aerorsync_metadata(
/// connection from the pool, so the cap stays conservative; the live
/// benchmark in master 9.6.2 says where it pays.
const SFTP_MULTI_THREAD_MAX_STREAMS: usize = 16;
/// Independent SSH connections a multi-file job may hold (see
/// `transfer_executor_max_sessions`).
const SFTP_POOL_MAX_SESSIONS: u16 = 16;

/// Job-wide guardrails for read-ahead. The byte budget includes one chunk per
/// reader, one channel window, and the writer's current chunk. The handle cap
Expand Down Expand Up @@ -2633,11 +2636,17 @@ impl StorageProvider for SftpProvider {
}
}

/// Conservative initial cap, mirroring the FTP pool clamp (1..8).
/// Each lease is a full independent SSH connection; raise only after a
/// live benchmark on the target server says it pays.
/// Ceiling of independent SSH connections one job may hold. Each lease is
/// a full connection. The former cap of 4 asked for "a live benchmark on
/// the target server" before being raised; the DAG engine review battery
/// on the Hetzner lab (wired gigabit, 5000 x 4 KiB files, 2026-09-08) is
/// that benchmark: at --parallel 16 rclone gained 1.9x on upload and 3.6x
/// on download over 4, while AeroFTP moved 1% because the cap silently
/// bound the flag. 16 matches rclone's range and the clamp already used by
/// --sftp-concurrency; the effective count stays min(ceiling, --parallel),
/// so the default of 4 connections is unchanged.
fn transfer_executor_max_sessions(&self) -> u16 {
4
SFTP_POOL_MAX_SESSIONS
}

/// Produce an independent transfer worker. It is **not connected**:
Expand Down Expand Up @@ -3970,6 +3979,30 @@ mod tests {
assert!(worker.supports_transfer_worker_reuse());
}

#[test]
fn sftp_pool_ceiling_matches_the_intra_file_stream_range() {
// --parallel is documented up to 32 and clamped per provider; the SFTP
// ceiling follows the same 16 as --sftp-concurrency, so a request of
// 16 is honoured and a request of 4 still yields 4 connections.
let config = SftpConfig {
host: "example.com".to_string(),
port: 22,
username: "testuser".to_string(),
password: Some(secrecy::SecretString::from("testpass".to_string())),
private_key_path: None,
key_passphrase: None,
initial_path: None,
timeout_secs: 30,
trust_unknown_hosts: false,
};
let provider = SftpProvider::new(config);
assert_eq!(provider.transfer_executor_max_sessions(), 16);
assert_eq!(
provider.transfer_executor_max_sessions() as usize,
SFTP_MULTI_THREAD_MAX_STREAMS
);
}

#[test]
fn sftp_workers_opt_into_warm_reuse_like_ftp() {
// PD-FTP-2 pool semantics: a worker is recycled only when the provider
Expand Down
Loading
Loading