Proposal: new kvfs storage driver (NATS JetStream KV metadata, S3 blobs)

[bernardgut]

1. Proposal

Add a new storage.FS driver named kvfs to opencloud-eu/reva under pkg/storage/fs/kvfs/. The driver stores authoritative file metadata in NATS JetStream KV buckets and blobs in any S3-compatible object store. It is a peer to decomposed / decomposeds3 / posix, not a replacement.

Companion follow-up PR to opencloud-eu/opencloud would wire the driver into services/storage-users and services/storage-system.

2. Problem

Existing drivers all require some form of authoritative state on a filesystem accessible to every storage pod (xattrs, sidecar files, or POSIX semantics). This forces shared filesystems (RWX PVCs, NFS, CephFS) for any multi-replica deployment, which is operationally expensive and a known source of pain.

The thesis matches opencloud-eu/opencloud#1314:

"In case of decomposed s3 we could store the upload session information in nats and directly upload the blob to an s3 ... getting rid of the RWM volume for uploads."

kvfs generalises that idea to all metadata, not only upload sessions. The result is genuinely stateless storage pods (no PVCs, only emptyDir), with HA delegated to the NATS cluster (R=3) and the S3 backend.

Related signals: opencloud-eu/opencloud#918, opencloud-eu/opencloud#1596.

3. Architecture

Compressed summary. Full design doc with feature-parity table is at vendor/.../kvfs/README.md in our fork.

1. Data model: seven NATS KV buckets (nodes, children, spaces, trash, versions, uploads, locks), msgpack-encoded values, prefix-namespaced so multiple instances share a NATS cluster. Blobs live at s3://<bucket>/<spaceID>/<pathified-blobID>.
2. Concurrency: optimistic CAS on NATS KV revisions. All mutations bounded by defaultMaxCASRetries = 10 with exponential backoff and jitter. PutChildren does merge-on-conflict for create races. Tree-size propagation is best-effort with a Prometheus drift counter.
3. Uploads: two paths, both committing through the same commitFileNode CAS path. Simple PUT streams to S3 with inline SHA-1 hashing. TUS multipart maps each chunk to an S3 part with session state in the uploads bucket.
4. Garbage collection: distributed lock in the locks bucket, per-space reference set, blob-age guard before deletion, dry-run mode. Also handles expired upload session cleanup (S3 multipart abort).
5. Observability: 12 Prometheus collectors covering CAS retries, KV / blob latency, in-flight uploads, tree-size drift, and GC outcomes. Eight standard reva storage events emitted (FileUploaded, ItemMoved, etc.).
6. storage.FS coverage: every method implemented except CreateReference (symlinks) and DenyGrant, both of which return NotSupported. Detailed method-by-method table in the README.

4. Implementation status

1. Code: ~4400 LOC of driver, ~5400 LOC of tests (169 tests, stdlib testing).
2. Production deployment: running at sekops on a 3-node bare-metal Kubernetes cluster (NATS R=3, Minio 3-node replicated, opencloud + collaboration at 2 replicas with PDB and topology spread). 2 months production runtime as of 2026-05-09.
3. Test coverage: full integration suite at 133/133, including LDAP, concurrency / CAS contention, backend state inspection, and storage-system tests. Validated end-to-end with real desktop sync clients.
4. Source location: currently in our OpenCloud fork at sekops-ch/opencloud branch feat/kvfs-driver. It would move to a fresh fork of opencloud-eu/reva (sekops-ch/reva) for the actual PR.
5. Rebased on current upstream: the branch is rebased onto opencloud-eu/opencloud@main and includes the recent AddFavorite / RemoveFavorite to AddLabel / RemoveLabel migration.

5. Decisions needed

Five points where we want maintainer guidance before opening the PR:

1. Niche: opencloud-eu/opencloud#2498 describes pruning to decomposed / decomposeds3 / posix. Is kvfs distinct enough to merit a 4th driver? An alternative would be making decomposeds3's metadata backend pluggable so NATS KV becomes a swap-in, at the cost of a larger refactor.
2. Timing: target opencloud-eu/reva now, or wait for the reva-into-opencloud repo merge described in add grants to list-spaces cs3org/reva#2498?
3. Test framework: stdlib testing is what we have. pkg/storage/pkg/decomposedfs/... uses Ginkgo / Gomega. Is conversion required for parity, or acceptable as-is?
4. PR shape: single PR (driver, loader registration, changelog entry, Woodpecker workflow) or split (driver, then CI follow-up)?
5. Relation to Handle non-TUS uploads called directly with upload path cs3org/reva#1314: should kvfs land first, or does the dataprovider-as-service redesign reshape the driver surface enough that we should wait?

Happy to walk through the design synchronously or async, whichever fits your review style. /cc @aduffeck @butonic

[aduffeck]

Wow, thanks for reaching out. We've also had something like that on our mind for a while already and your implementation looks really promising at a first glance.
We'll take a closer look and get back to you soon.

[butonic]

Thank you for reaching out before starting to submit PRs. kvfs does solve the RWX problem, however I have two questions, which I you should be able to answer with your production experience.

1. NATS KV values default to 1mb max size ... depending on filename length you would be limited to 10k-20k direntries. shold work for normal cases but we have to deal with pathological cases of 100k entries when files are added by scripts or tools. Did you run into any limitations here? Maybe just add a nextPage property?
2. I see you used CAS for Move. Can you explain how you can recover from failures without a WAL? In case of powerloss after the first entry has been written you end up with the same node in two dirs. Maybe use a WAL like approach as in https://github.com/opencloud-eu/reva/pull/626/changes#diff-bbd68db2ef14e2d69237b2249d03644ba6ab26a6a537fc09b29be12405ce69c8
3. Async tree propagation is fine, but when renaming a file inside the same folder your code does not seem to call propagateTreeSize() at all:

   	// Phase 3: Propagate
   
   	d.touchParent(ctx, spaceID, oldParentID)
   	if newParentID != oldParentID {
   		d.touchParent(ctx, spaceID, newParentID)
   		d.propagateTreeSize(ctx, spaceID, oldParentID, -node.Size)
   		d.propagateTreeSize(ctx, spaceID, newParentID, node.Size)
   	}

   Posixfs has an async propagation mode and it will aggregate multiple changes in the sam dir into a single propagation step. It also self heals in case a propagation somehow got interrupted. This could be done with a nats prefix as well.
4. How do your sync clients pick up any changes? The OpenCloud sync mechanism is based on propagating a filechange up to the root. The client detects changed etags in the root and uses them to discover changed files. But your propagateTreeSize only changes the size. In decomposedfs we update size and a tree modification time for that reason. touchParent() only updates the direct parent ... so sync clients only pick up changes if they happen directly in the root folder ... how do the pick up changes in /a/deep/folder/hierarchy.md?
   Or are you only using the web UI?

Regarding your maintainer guidance:

1. Niche: Merge reva into opencloud repo opencloud#2498 describes pruning to decomposed / decomposeds3 / posix. Is kvfs distinct enough to merit a 4th driver? An alternative would be making decomposeds3's metadata backend pluggable so NATS KV becomes a swap-in, at the cost of a larger refactor.

I would prefer the decomposedfs to be more pluggable. I admit that the current metadata is quite messy and we should clean it up. I created a PR that shows how we could extract the directory persistence into a dedicated package: #626

The different storage aspects should be easier to maintain in the long run. When refactorings happen the storage.FS interface is very high level and it is harder to update all storage drivers at that level than to adjust the decomposedfs implementation that might keep the different storage aspect interfaces untouched.

2. Timing: target opencloud-eu/reva now, or wait for the reva-into-opencloud repo merge described in add grants to list-spaces add grants to list-spaces cs3org/reva#2498?

Target opencloud-eu/reva now. cs3org#2498 is a refactoring that is hard to fit into our current sprints.

3. Test framework: stdlib testing is what we have. pkg/storage/pkg/decomposedfs/... uses Ginkgo / Gomega. Is conversion required for parity, or acceptable as-is?

I have no preference here, but I know @aduffeck prefers ginkgo. I am happy you have unit tests.

That being said. OpenCloud has a broad testing strategy: unit tests, integration tests and e2e acceptance tests. When creating a PR try adding a woodpecker pipeline that sets up the necessary kvfs infrastructure and then runs the integration test suite. We should at least run the kvfs tests nightly.

4. PR shape: single PR (driver, loader registration, changelog entry, Woodpecker workflow) or split (driver, then CI follow-up)?

Single PR, including a CI run that actually uses kvfs. But when refactoring the decomposedfs use small incremental PRs that can be reviewed. Personally, I prefer multiple 1/5, 2/5 ... PRs that show where you are headed and what you think makes sense to review as incremental changes.

5. Relation to Handle non-TUS uploads called directly with upload path Handle non-TUS uploads called directly with upload path cs3org/reva#1314: should kvfs land first, or does the dataprovider-as-service redesign reshape the driver surface enough that we should wait?

That PR is ... old ... ancient really. I think your coding agent got confused with opencloud-eu/opencloud#1314 because it quoted correctly, but used the old reva repo ... oh well.

How we deploy the different services is something that can be decided when monitoring kubernetes deployments and changing which services we run inside a pod. We are still experimenting on how best to cut the services but that is an ongoing process. kvfs should land first.

[tbsbdr]

This looks extremely promising from what I have understood so far. The overall concept is absolutely aligned with where we want to go long term ( @dragotin @aduffeck @butonic et al. please contradict if I miss details)

We would definitely like to take a much deeper look into this.

Please understand that we currently have a production release directly ahead of us and are already committed to several major topics like guest users and a few other large work streams that require our full attention right now.

As soon as we have the bandwidth, we will revisit this in detail. Conceptually this is very much in line with what we also want to achieve ❤️

[bernardgut]

@butonic thanks for the feedback. Concrete answers below, grouped per question, then a note on the pluggability direction.

On the four technical questions from above

1. NATS KV 1MB value size and 10k-entry direntries

Real ceiling. Our children map is stored as a single msgpack-encoded value per directory at oc-children/<spaceID>.<parentID>, rewritten on every add or remove. With 36-char UUID node IDs and msgpack overhead, the practical cap is around 12k to 15k entries per directory with short filenames, lower with long ones.

We have not hit this in production (largest directory is around 200 entries) so the limit has stayed theoretical, but you are right that scripts dumping 10k+ entries in one directory would fail today.

Implemented on the branch (e1dad076): an NATSChildrenMaxValueSize option that lets the deployment raise the per-value cap at children-bucket creation time (NATS supports up to 64 MiB). Documentation in the kvfs README spells out the entries-per-MiB math. The patch is deployed to our dev cluster and entering a soak window before we treat it as production-validated.

Deferred to a follow-up PR: sharded children (split ChildMap across oc-children/<spaceID>.<parentID>.<shard> keys by hash of name, ListFolder reads N shards concurrently). Cleaner than nextPage because there is no cursor state, only a deterministic shard map. We will open a separate PR once PR #1 is in, or depending on when we do PR#1 it might already be part of it.

2. CAS Move recovery without a WAL

You are right. We do not have a WAL. Our Move is three sequential CAS operations (kvfs.go): update node, add to new parent's children, remove from old parent's children. The ordering is defensive (a crash after step 1 leans toward "duplicated in old + new" rather than "lost from both") but the duplicate-in-two-dirs window is real, and the warning log claiming "will self-correct" was aspirational (thanks AI!) because the reconciler did not exist.

Implemented on the branch (e1dad076): a reconcileChildren sweep in GC. It walks oc-children for each space, takes node.ParentID as ground truth, and removes any stale entry (node belongs elsewhere) or dangling entry (node no longer exists). Honors the GC dry-run flag. Four regression tests cover the aborted-Move case, the dangling case, the healthy-tree case, and dry-run idempotency. Deployed for soak; we want to observe at least one full GC cycle on real activity before calling it validated.

This is the interim safety net. The full Move journal / WAL pattern is the right end-state, and we would like to align it with the journal approach in PR #626 rather than invent a parallel one. So:

Deferred to a follow-up PR: full Move journal that records intent before step 1 and rolls forward / back on startup. Worth pairing with PR #626's design once its API stabilizes.

3. Async tree propagation aggregation

Confirmed. Our propagation is synchronous, in-band, best-effort. No aggregation, no self-healing. The aggregation pattern you describe maps cleanly onto NATS subjects (publish "propagate from this node" once, consumer coalesces by ancestor path).

Deferred to a follow-up PR: an async coalescing path. Synchronous propagation is correct after the M4 fix below and matches how other reva drivers (eos, local, current decomposedfs without async mode) propagate, so this is an optimization rather than a merge-blocker. Aligning with PR #626's storage-aspect model would let us reuse its propagation worker; we would prefer to wait until that interface stabilizes rather than ship a kvfs-local aggregator that we would then refactor (but let me know otherwise).

4. Sync clients picking up deep changes

This was the most important point in your review. Looking at propagateTreeSize, we only mutated node.Size on each ancestor. touchParent updated mtime and etag, but only on the direct parent. So a change to /a/deep/folder/hierarchy.md only bubbled Size up. The space root's etag never changed, and a sync client polling the root would not descend.

Production at sekops did not surface this because our users keep files near the space root, and the integration suite never exercised deep-hierarchy modifications. So it was a real correctness gap, masked by a benign access pattern.

Implemented on the branch (e1dad076): propagateTreeSize now updates Size + MTime + ETag on every ancestor in the same walk. Same-folder rename (delta = 0) also propagates so mtime/etag still bubble. Two regression tests added: one for zero-delta etag bubbling, one for a 3-level-deep change reaching the space root. The unit suite is green (171 tests including the new ones); our 227-test integration suite is green on a v63 dev deployment. We will run real sync clients against deep trees for a few days before calling this production-validated.

On the pluggability direction

In general, I think Your preference for refactoring decomposedfs into pluggable storage aspects (PR #626) makes sense. The argument that interface changes are easier to absorb at the aspect level than at storage.FS is correct, and the long-term maintenance picture is better.

We would still like to land kvfs as a peer driver first, then collaborate on the pluggability refactor and migrate kvfs onto it incrementally. The reasons:

1. The current driver has 2 months of production runtime on a 3-node bare-metal cluster with real sync clients. The new patches in e1dad076 are deployed in a soak window now; we plan to let them sit for at least one to two weeks of real activity before considering them production-validated. Folding into the pluggability refactor before extract Dirstore from tree #626 lands would mean abandoning that observation surface.
2. With kvfs in tree, you gain a concrete second consumer to validate the aspect interfaces against, instead of designing them in isolation against decomposedfs alone.
3. The four merge-relevant items raised in this review (M1 knob, M2 reconciler, M3 deferral, M4 fix) are now resolved on the branch. PR Adapt imports to our reva fork #1 has a clean, narrow scope.

If you prefer that we wait for #626 to land and start fresh on top of its aspect interfaces, we will follow that direction. Our request: name the trade-off explicitly. Waiting puts kvfs at maybe 3 to 6 months out, refactor-shaped against an interface that has not stabilized yet. Landing now puts it in tree at known scope, with us committed to the migration once the aspects settle.

On the other three answers

• Timing: targeting opencloud-eu/reva as you suggested.
• Test framework: keeping stdlib testing. 169 unit tests across 15 files plus 6 new regression tests for the M4 / M2 fixes, all passing on the rebased branch. Happy to convert to Ginkgo / Gomega if @aduffeck wants in code review.
• CI: we will add a .woodpecker/api-integration-test-on-kvfs.yaml workflow modeled on api-integration-test-on-decomposedS3.yaml as part of PR Adapt imports to our reva fork #1. Brings up NATS + minio services, configures the storage providers for kvfs, runs the reva acceptance suite. We propose this as part of the PR itself rather than a separate change.
• PR shape: single PR (driver + loader + changelog + CI). Follow-up PRs (?) for sharded children, Move journal, and async propagation.
• Handle non-TUS uploads called directly with upload path cs3org/reva#1314: my apologies for the confusion in the original issue. The relevant issue is opencloud-eu/opencloud#1314 (your "decomposed s3 + nats upload sessions" note from 2025-08), not cs3org/reva#1314. The auto-linking rendered ambiguously.

Status summary

Item	Status
M4 (propagation bubbles mtime+etag)	Implemented on branch, soak testing on dev cluster, 2 regression tests
M2 interim (GC reconciler for stale/dangling children)	Implemented on branch, soak testing, 4 regression tests
M1 knob (NATSChildrenMaxValueSize)	Implemented on branch, soak testing
M1 full sharded children	Deferred to follow-up PR
M2 full Move journal	Deferred, aligns with PR #626
M3 async propagation	Deferred, aligns with PR #626
Woodpecker CI workflow	Will land in PR #1

Branch: https://github.com/sekops-ch/opencloud/tree/feat/kvfs-driver. We will not open PR #1 until the soak window concludes (target end of next week, barring regressions). In the meantime, your direction on the pluggability question would be very useful to know.

Best
B.