Add can-list-ordered capability flag

[damooo]

Few backends like S3 explicitly returns alphabetically ascending ordered items in their list apis, as public api. And backends like rust-fs explicitly mentions order is arbitrary.

There are few invariants one can use for ordered backends. Hece it will be helpful to have a capability flag in accessor metadata.

[Xuanwo]

There are few invariants one can use for ordered backends.

Can you explain a bit? What's the benefit if the backends return ordered items? (NOTE: s3 returns item in utf-8 binary order).

[damooo]

Then we can check, if a related object exists in streaming way.

Say, i have an object abc.json, and it's actual content-type is application/ld+json, and not just plain application/json, that can be decoded from extension using mimeguess. Then we are adding another object called abc.json$meta_overrider, which contains overrided metadata. Now we can know if such an overrider exists or not, in streaming way by buffering upto paths that str-compare less-than-or-eq to overrider path. Othewise, we have to load entire list to know resolve effective values.

[Xuanwo]

Now we can know if such an overrider exists or not, in streaming way by buffering upto paths that str-compare less-than-or-eq to overrider path. Othewise, we have to load entire list to know resolve effective values.

What's the different with stat("abc.json$meta_overrider")?

The thing you prefer is the following?

if op.can_list_ordered() {
   check_by_path()
} else {
   check_by_stat()
}

[damooo]

Yes, when listing a dir, one can know if an overrider exists or not in streaming way, with out making is_exists call for overrider corresponding to each file. And read overrider only for very small number of matches.

[Xuanwo]

We need something like AccessorHints to give users hints about its internal behavior. But I don't want to introduce this feature so quickly. Let's wait and see if our community has similar/common needs for this too.

At the same time, we plan to

• Implement metadata cache for Object #605
• Reuse the metadata during the list.

They will make the is_exists calls much cheap.

[damooo]

Indeed. Such AccessorHints, documenting invariants corresponding to public apis of different backends will be much useful. So that we use abstraction layer for much of the part, and can optionally depend on backend specific capabilities to optimize for some majority cases.

Metadata cache sounds great. But sometimes, one may have to depend only on live metadata, if object store is being used by different nodes in a cluster in their own processes. May be ability to plug external cache, that can also stat/list efficiently like redis will be useful in such cases.