Proposal: Add `ObjectSpaceKind` to `AccessorMetadata`

[damooo]

Service backends fundamentally vary in organizing their object space.

Backends like fs, solid has Hierarchical object space. Where as S3, and major cloud object stores have Flat object space. This different leads to different semantics over operations on objects in them, like following:

• / has special meaning in hierarchical space, that represents an hierarchical level. where as in flat space it doesn't have any semantics.
• One can create a/b/c with out creating a/, a/b/ in flat space, where as in hierarchical space we cannot. (Thus current implementation of fs service creates a/, a/b/ `implicitly.)
• One can have a/, a paths in flat space, where as that is not the case in hierarchical space.
• One cannot delete a/b/ with out deleting a/b/c in case of hierarchical space. Where as in flat space, one can.
• etc.

These subtle differences diverges expected layout. For applications that rely on predictable layouts, it creates serious errors.

Hence proposal is to have a object_space_kind property with an enum in AccessorMetadata. So that applications can customize control flow based on that if required.

[Xuanwo]

Thanks for your proposal, this is an exciting idea!

I agree with your point that:

Service backends fundamentally vary in organizing their object space.

This is exactly why OpenDAL is built up: Access data freely, painless, and efficiently

We take freely as the first because it's the most important: OpenDAL must be able to access different storage systems in the same way.

So we have the following undocumented rules:

• OpenDAL has a carefully picked behavior set, for example:
  • DELETE must return OK(()) no matter the file exists or not
  • WRITE a/b/c must create a/b before writing.
• OpenDAL also has an addable capability, like presign and multipart
  • Like rust's feature design, users can only decide to use it or not but can't choose different behavior for the same operation.

---

Let's come back to this proposal.

• / has special meaning in hierarchical space, that represents an hierarchical level. where as in flat space it doesn't have any semantics.

OpenDAL will treat / as dir even for object storage systems like s3 and azblob.

• One can create a/b/c with out creating a/, a/b/ in flat space, where as in hierarchical space we cannot. (Thus current implementation of fs service creates a/, a/b/ `implicitly.)

That's true. We will add this as a noted behavior for the write operation.

• One can have a/, a paths in flat space, where as that is not the case in hierarchical space.

That's true. OpenDAL decides not to support this case hence nearly no one depends on this behavior.

We can add this into behavior too.

• One cannot delete a/b/ with out deleting a/b/c in case of hierarchical space. Where as in flat space, one can.

That's true. OpenDAL should clarify that DELETE only deletes one file or an empty dir.

NOTE: no storage system supports this even on fs since there is no such syscall for rm -rf. We prefer to expose a low-level API and build high-level APIs upon this like BatchOperator.

So that applications can customize control flow based on that if required.

As described in the previous introduction, customize control flow on different layouts should be avoided in OpenDAL. Users should always write the same logic while using OpenDAL.

However, OpenDAL is allowed to add new features like can_list_prefix.

---

We will clarify the detailed behavior for every operation.

[damooo]

I think my intention is misunderstood.

Proposal is not to fragment opendal layer based on backend. OpenDAL will have exactly same api and semantics for all backends.

The things i listed are not the features along which i seek fragmentation. Instead they are issues faced when trying for single abstraction layer. Those cannot be mitigated by OpenDAL.

Thus, OpenDAL cannot create a/b/c with out creating a/b/ in h-space. But application cannot determine that from Operator / dyn Accessor. So currently application cannot ensure required invariants.

Same-way, with OpenDAL, creating a/b/ will succeed in f-space, eve if there is a/b . Where as in h-space it will error. Application cannot know that beforehand to customize control flow.

For such cases, proposal is to provide a flag indicating what is the kind backend obj-space. I emphasize on backend-space not the space exposed by OpenDAL. I.e. to expose the property of backend for ensuring further guarantees at application layer, which OpenDAL or any other layer cannot generically ensure for all backends.

[Xuanwo]

Thus, OpenDAL cannot create a/b/c with out creating a/b/ in h-space. But application cannot determine that from Operator / dyn Accessor. So currently application cannot ensure required invariants.

So the problem here is you want to get a NotFound error if the parent dir is not existed instead of creating them automatically?

Same-way, with OpenDAL, creating a/b/ will succeed in f-space, eve if there is a/b . Where as in h-space it will error. Application cannot know that beforehand to customize control flow.

So does this. Do you expect to get an error if a/b exists before creating a/b/?