Bug when using in_memory_cache

[snwagh]

Description

A nasty bug in local testing. The bug occurs when we do not use the in_memory_cache=False when creating connections:

ds_client, do_client = SyftboxManager.pair_with_in_memory_connection(
    sync_automatically=False, use_in_memory_cache=False
)

The error shows up as dataset changes not percolating from DO to DS.

[prabindersinghh]

Hi! I’d like to take this issue up if that’s okay.

I’ll first try to reproduce this with use_in_memory_cache=False, identify where dataset state is not being refreshed between DO → DS, and then propose a fix with a regression test so this doesn’t reoccur.

Will update here once I have a clear root cause.

[prabindersinghh]

Thanks for assigning this!

From the description it looks like the sync works with use_in_memory_cache=True but fails when persistence is used.

I suspect the non-memory storage backend is not updating the shared state during sync() — possibly the dataset registry is written locally but not propagated to the paired client.

I’ll first trace the sync path from pair_with_in_memory_connection() → sync() → dataset registry update to understand where the divergence happens.

I’ll report back once I narrow down the code path 👍

[prabindersinghh]

I investigated this further and it seems the issue is not directly caused by the cache layer, but by dataset discovery during incremental sync.

When use_in_memory_cache=False, datasets arrive across multiple sync events:

1. dataset directory created
2. partial contents synced
3. dataset.yaml synced last

During step (2), SyftDatasetManager.get_all() attempts to load the dataset directory.
Since metadata is not yet present, _load_dataset_from_dir() raises, and the directory is skipped.

Because persistent mode does not rebuild state on every run, the dataset is never retried and remains invisible on the DS side.

This also explains why use_in_memory_cache=True works, the state is reconstructed each run so the dataset is eventually discovered.

From what I can see, discovery currently relies on catching exceptions:

try:
dataset = self._load_dataset_from_dir(dataset_dir)
all_datasets.append(dataset)
except Exception:
continue

So a directory scanned before metadata exists is effectively ignored permanently.

It seems dataset discovery should instead be idempotent, only treat directories containing dataset.yaml as datasets and allow them to appear once metadata arrives.

I tested locally by checking for metadata before loading, and sync worked correctly in persistent mode.

Does this match the intended behavior of the sync/discovery system?
If so, I can open a PR with a minimal change.