`Storage::delete` returns immediately without awaiting completion, causing stale reads

[Barre]

I have a use case where I need to guarantee that the cache is invalidated after calling Storage::delete. Currently, this method appears to return immediately without providing a way to await the actual deletion completion.

Current Behavior:

• delete() returns immediately
• The actual deletion happens asynchronously in the background
• Other tasks can read stale cache entries even after delete() has been called

Expected Behavior:

• Either delete() should be async and complete when the entry is fully removed
• Or there should be a way to await/confirm the deletion has completed
• No stale reads should be possible after a successful delete operation

[MrCroxx]

Hi, @Barre . Thank you for reporting this issue.

IIRC, you are developing ZeroFS on SlateDB, both of the systems are built on S3. In this case, as a single-machine cache, ensuring the consistency of deletes is very difficult.

Challenges:

• A single-machine cache cannot perceive whether another node has deleted an object on its own.
• When the hybrid cache is based on recovery from disk, it cannot guarantee whether other nodes have updated or deleted objects in the meantime on its own.
• Tracking the completion of deletes requires significant additional overhead, and most S3-based systems do not require strict deletion.

Therefore, when deleting in Foyer, only the memory index of the object in the in-memory cache and disk cache will be deleted, and it will be removed from the disk asynchronously. This approach ensures delete consistency when recovery does not occur or when the recover mode is configured as None.

[MrCroxx]

Foyer has reserved a tombstone log implementation to ensure strict deletion. However, this implementation is also being refactored to ensure better support.

Strict consistency guarantees are still in the development plan for Foyer; for now, Foyer is more suitable for use in systems like S3 that do not have updates.

[Barre]

Thanks you for your reply.

Foyer has reserved a tombstone log implementation to ensure strict deletion. However, this implementation is also being refactored to ensure better support.

Strict consistency guarantees are still in the development plan for Foyer; for now, Foyer is more suitable for use in systems like S3 that do not have updates.

Do the "classic" Memory Cache ensures strict deletion?

[MrCroxx]

Do the "classic" Memory Cache ensures strict deletion?

Sure! It is always safe with in-memory cache, and even with disk cache without recovery (set recover mode to None).

[MrCroxx]

In fact, I have also asked the same question to the CacheLib project. CacheLib also cannot guarantee the consistency of deletes during recovery in a non-graceful shutdown (and even CacheLib's method may not be able to recover during a non-graceful shutdown, which is related to its trade-offs).

I hope Foyer can do better in this regard, but it will take more time to implement, experiment and benchmark. The tombstone log is one such attempt.

[Barre]

Do the "classic" Memory Cache ensures strict deletion?

Sure! It is always safe with in-memory cache, and even with disk cache without recovery (set recover mode to None).

Oh, I don't really mind about recovery, that's great, I'll go that route then! Thanks.

[Barre]

I tried this just now

https://github.com/Barre/ZeroFS/pull/65/files#diff-5caf3cc5d3186459b336694e55ad1898aaf48294956d7886e318cde96bb9eb99R79

I get a bunch of corruption that way, so it seems that replacing / deleting is not immediate even with .with_recover_mode(foyer::RecoverMode::None)

Eg.

Cloning a git repo with a ZeroFS mount:

/mnt3# git clone "https://github.com/slatedb/slatedb.git"
Cloning into 'slatedb'...
remote: Enumerating objects: 5176, done.
remote: Counting objects: 100% (1818/1818), done.
remote: Compressing objects: 100% (422/422), done.
remote: Total 5176 (delta 1656), reused 1422 (delta 1395), pack-reused 3358 (from 2)
Receiving objects: 100% (5176/5176), 2.39 MiB | 915.00 KiB/s, done.
Resolving deltas: 100% (3697/3697), done.
fatal: bad object 6126c0902870c7cf028e381903fd02678e0edffb
fatal: remote did not send all necessary objects

Compiling a rust program

error: the compiler unexpectedly panicked. this is a bug.
error: the compiler unexpectedly panicked. this is a bug.


note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md
note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md


note: please make sure that you have updated to the latest nightly
note: please make sure that you have updated to the latest nightly


note: please attach the file at `/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anstyle-parse-0.2.6/rustc-ice-2025-08-04T15_22_02-1578036.txt` to your bug report
note: please attach the file at `/root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/parking_lot_core-0.9.10/rustc-ice-2025-08-04T15_22_02-1578034.txt` to your bug report


note: compiler flags: --crate-type lib -C embed-bitcode=no -C debuginfo=2
note: compiler flags: --crate-type lib -C embed-bitcode=no -C debuginfo=2

Everything is fine with the in-memory only cache.

[Barre]

Additionally, this check never completed in my PR https://github.com/Barre/ZeroFS/actions/runs/16726521852/job/47343509451?pr=65

Most probably indicating corruption there too.

[MrCroxx]

Do worry. Let me take a look. I'm working on a hugo refactor, my bandwidth is limited. Maybe a little bit late.