Warning and incorrect behavior on file listing with concurrent read loads

[gaberudy]

Mountpoint for Amazon S3 version

mount-s3 v1.20.0

AWS Region

us-east-1

Describe the running environment

AWS m6i.2xlarge running mount-s3, scheduled jobs from remote machines are connecting via a sshfs mount of the server that has access to the mount-s3 folder.

Mountpoint options

/usr/bin/mount-s3 -f --uid 166535 --gid 166535 --write-part-size 16777216 --force-path-style --allow-other --allow-delete --allow-overwrite demo-prod-gh-bucket "/opt/ghdata/share/CloudStorage"

What happened?

Using mount-s3 in a bioinformatic workflow with ~ 700 files and folders created and used in a directory.

I'm doing work in batches of 10 workers on a host. In the latest v1.20.0 release, I can reproduce a behavior where, in a queue of 100 workers, on the 8th or 9th batch, I get errors seeing the files in the work directory. A program reports that a file doesn't exist, but it does. Sometimes it's recovered by the 9th or 10th batch. I suspect it has something to do with the number of files being over a page size in the API or something.

Note that downgrading to 1.14.0 (last known good version).

I tested downgrading to 1.19.0 and v1.14.0 and was able to reproduce the issue, although the messages where different.

It seems like some cached or intermediate value for the directory listing is out of sync with the folder (as files are added). The problem "fixes" itself, but results in jailed tasks / jobs as they see a truncated or incomplete listing for a directory.

Any suggestions or work-arounds or coudl mount-s3 auto-resolve this state inconsistency internally?

Relevant log output

Logs at around the time of the error (1.20 and 1.19)
---
[WARN] ThreadId(12) readdirplus{req=52422080 ino=2711 fh=5801 offset=113}: mountpoint_s3_fs::fuse: readdirplus failed with errno 9: inode error: out-of-order readdir, expected offset 118 but got 113 on dir handle 5801
2025-09-22T17:25:45.807592Z  WARN ThreadId(12) readdirplus{req=52422080 ino=2711 fh=5801 offset=113}: mountpoint_s3_fs::fuse: readdirplus failed with errno 9: inode error: out-of-order readdir, expected offset 118 but got 113 on dir handle 5801
[WARN] ThreadId(14) readdirplus{req=52422182 ino=2711 fh=5801 offset=113}: mountpoint_s3_fs::fuse: readdirplus failed with errno 9: inode error: out-of-order readdir, expected offset 118 but got 113 on dir handle 5801
2025-09-22T17:25:45.822044Z  WARN ThreadId(14) readdirplus{req=52422182 ino=2711 fh=5801 offset=113}: mountpoint_s3_fs::fuse: readdirplus failed with errno 9: inode error: out-of-order readdir, expected offset 118 but got 113 on dir handle 5801
[WARN] ThreadId(13) readdirplus{req=52422188 ino=2711 fh=5801 offset=113}: mountpoint_s3_fs::fuse: readdirplus failed with errno 9: inode error: out-of-order readdir, expected offset 118 but got 113 on dir handle 5801
2025-09-22T17:25:45.822259Z  WARN ThreadId(13) readdirplus{req=52422188 ino=2711 fh=5801 offset=113}: mountpoint_s3_fs::fuse: readdirplus failed with errno 9: inode error: out-of-order readdir, expected offset 118 but got 113 on dir handle 5801
[WARN] ThreadId(10) readdirplus{req=52422190 ino=2711 fh=5801 offset=113}: mountpoint_s3_fs::fuse: readdirplus failed with errno 9: inode error: out-of-order readdir, expected offset 118 but got 113 on dir handle 5801
2025-09-22T17:25:45.822361Z  WARN ThreadId(10) readdirplus{req=52422190 ino=2711 fh=5801 offset=113}: mountpoint_s3_fs::fuse: readdirplus failed with errno 9: inode error: out-of-order readdir, expected offset 118 but got 113 on dir handle 5801
[WARN] ThreadId(08) readdirplus{req=52422474 ino=2711 fh=5800 offset=113}: mountpoint_s3_fs::fuse: readdirplus failed with errno 2: inode error: file "C0749-2-3-4_dnascope_run.log" does not exist in parent inode 2711 (partial key "Output/Panel 100 Run2/")
2025-09-22T17:25:45.854386Z  WARN ThreadId(08) readdirplus{req=52422474 ino=2711 fh=5800 offset=113}: mountpoint_s3_fs::fuse: readdirplus failed with errno 2: inode error: file "C0749-2-3-4_dnascope_run.log" does not exist in parent inode 2711 (partial key "Output/Panel 100 Run2/")

Example messages in 1.14:
---
2025-09-22T20:21:24.717444Z  WARN readdirplus{req=5681170 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402
2025-09-22T20:21:24.717505Z  WARN readdirplus{req=5681172 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402
2025-09-22T20:21:24.717558Z  WARN readdirplus{req=5681174 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402
[WARN] readdirplus{req=5681170 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402
[WARN] readdirplus{req=5681172 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402
[WARN] readdirplus{req=5681174 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402
[WARN] readdirplus{req=5681182 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402
2025-09-22T20:21:24.717940Z  WARN readdirplus{req=5681182 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402
[WARN] readdirplus{req=5681184 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402
2025-09-22T20:21:24.717995Z  WARN readdirplus{req=5681184 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402
[WARN] readdirplus{req=5681188 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402
2025-09-22T20:21:24.718174Z  WARN readdirplus{req=5681188 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402
2025-09-22T20:21:24.718232Z  WARN readdirplus{req=5681190 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402
[WARN] readdirplus{req=5681190 ino=3561 fh=4599 offset=402}: mountpoint_s3::fuse: readdirplus failed with errno 22: out-of-order readdir, expected=416, actual=402

[tadiwa-aizen]

Hello, thank you for raising this issue!

First, we'd like some clarification on the versions. We see that both v1.14 and v1.20 show out-of-order readdir warnings in your logs. However, did v1.14 also experience the actual task failures where your bioinformatics programs reported files not existing, or were there only warnings without task impact?

Your logs show an offset-related error during directory listing operations. This is a current limitation of how Mountpoint handles directory access, however, it's unclear if these offset errors alone are causing your task failures.

The more concerning error appears in your v1.20 and v1.19 logs:

[WARN] ThreadId(08) readdirplus{req=52422474 ino=2711 fh=5800 offset=113}: mountpoint_s3_fs::fuse: readdirplus failed with errno 2: inode error: file "C0749-2-3-4_dnascope_run.log" does not exist in parent inode 2711 (partial key "Output/Panel 100 Run2/")

This appears to be a genuine issue in how Mountpoint handles adding file while listing.

We will investigate this further and return with a more detailed analysis and a potential solution.

[gaberudy]

Thanks for the follow-up and insight.

Even with v1.14 we were seeing the errors where the script thought a file did not exist that did exist (re-running would find the missing file).

A command like cp $output/HIS0049-NEG-3_deduped.bam* . might fail like:

cp: cannot stat '/w/workspace/CloudStorage/Output/Panel 100 Run 4/HIS0049-NEG-3_deduped.bam*': No such file or directory

I look forward to hearing how your analysis goes, let me know if I can provide any more context or details.