🪣 Support for OBject Storage

[Kezzsim]

Resolves #905

[danielballan]

The only remaining big thing here is to run a MinIO container on GHA so we can test against it.

[danielballan]

This ambiguity in return type could be a signal that the design isn't quite right.

If the problem is, "How can we get the path back to the user?" what are our other options? What would the consequences be of adding a path attribute to ObjectStorage?

[danielballan]

This works but it's a lot to digest:

• regex
• string emptiness check
• lstrip
• replace

Above, we have the parsed URL urlparse(uri), which we use to get the scheme. If we keep a reference to that in some local variable, parsed_uri, we can easily grab the path:

full_path = parsed_uri.path  # includes bucket and the rest
bucket_name, blob_path = full_path.split("/", 1)

Aside: If you haven't seen it before, the second argument to split works like this:

>>> 'a/b/c'.split('/')
['a', 'b', 'c']
>>> 'a/b/c'.split('/', 1)  # split on the first instance of '/' only
['a', 'b/c']

Now, to get the "bucket only" version of the URL:

from urllib.parse import urlunparse

# Get the URL without the blob_path, just the bucket.
bucket_uri = urlunparse(parsed_uri._replace(path='/' + bucket_name))
# Look up storage by bucket URI.
storage = _STORAGE[bucket_uri]
# Return a copy encapsulating the blob_path.
return storage.with_blob_path(blob_path)  # Note: This method would need to be added to ObjectStorage.

Example:

>>> uri = 'https://example.com/bucket/a/b/c'
>>> parsed_uri = urlparse(uri)
>>> parsed_uri
ParseResult(scheme='https', netloc='example.com', path='/bucket/a/b/c', params='', query='', fragment='')

>>> full_path = parsed_uri.path
>>> full_path
'/bucket/a/b/c'

>>> full_path.lstrip('/')
'bucket/a/b/c'

>>> bucket_name, blob_path = full_path.lstrip('/').split('/', 1)
>>> bucket_name
'bucket'
>>> blob_path
'a/b/c'

[Kezzsim]

Pertaining to the minio testing container:

• Specify Object fixture in test_writing.py
• Set fixture to xfail when minio url isn't available
• Add GitHub Actions to setup the minio container
• Add code to reset the state of the bucket after each test

[danielballan]

I like that we leave Azure and Google off now, until they can be tested, but we have a clear path for adding support soon.

https://github.com/bluesky/tiled/pull/1021/files#diff-38e7f2525a2fda64425a32d8b78379eedfbb2a2d3bd147e5c6f0571be0210201R29

And I like the clarity of this:

https://github.com/bluesky/tiled/pull/1021/files#diff-1fb9039deb7e0ac14eb3afa15f93579ebeef9858ca8275a7ccef55ad6b52277eR218-R219

I hope to test drive this before clicking the green merge button, but this will go in in time for maint, barring any surprises. Well done!