ListObjects: Cursor issue with prefix-like objects

[amwolff]

I found an API limitation of ListObjects that might cause issues for software that creates objects disguised as prefixes (this is common for software that talks to S3). If you upload:

• object
• object/ (this is not a prefix; it's an object)
• object/object

and then call ListObjects with {prefix=object/, recursive=true, limit=1} (or if object/ is the last element on the list page), it's impossible to go to the next page. libuplink expects Cursor to be stripped out of a prefix, so if you have object/ as a prefix and it's also the last object on the page, you are going to call ListObjects with Cursor="", which effectively puts you in an infinite loop.

[egonelbre]

It does sound like we need to differentiate between a nil and empty string, both on the protocol level, uplink API level and also probably in metabase.

Alternatively, a way to retrofit into the current API is to add a boolean flag that "cursor" is full path.

[amwolff]

I like the retrofit idea since the current API works for all cases except for this one.

[egonelbre]

Ok, let's add a AbsoluteCursor bool to retrofit the implementation. Respectively the uplink API and protocol both should then carry the boolean flag throughout the system.

[egonelbre]

While trying to implement this, I realized there's a slight problem with the "AbsoluteCursor" concept. It introduces this annoying need to juggle AbsoluteCursor: true and AbsoluteCursor: false. For example, if you want to start listing the prefix, you would need to specify Prefix: X, Cursor: "", AbsoluteCursor: false and then to continue listing, you would need to use Prefix: X, Cursor: X, AbsoluteCursor: true.

And while looking at the code I remembered we used to have something that would solve it as well: ListDirection_FORWARD and ListDirection_AFTER.

[shaupt131]

Not critical at the moment. Deprioritizing.