🪣 Support for Blob Storage

[Kezzsim]

Created to resoslve #905
Blob (Binary Large Objects) is a recent cloud native data format with many advancements over traditional file systems which allows it to handle larger volumes of data in containers with less stringent size limitations.

Zarr has added support for blob storage through Fsspec, and conversely through s3fs, a library which more succinctly implements cloud based connections.

This PR creates a blob storage dataclass for tiled, which alongside filesystem and sql will provide a location where data can be written to and read from. Unlike the presently available storage options, Blobs require uri, key and secret in order to connect to a datasource, omitting the last two parameters if said datasource is public.

[danielballan]

In the old model, the adapter was handed multiple storage options and had to pick the one it wanted. Now, the caller in tiled.catalog.adapter picks one storage option and passes just that one in.

tiled/tiled/catalog/adapter.py

Lines 672 to 695 in f3331ef

	adapter = STORAGE_ADAPTERS_BY_MIMETYPE[data_source.mimetype]
	# Choose writable storage. Use the first writable storage item
	# with a scheme that is supported by this adapter. # For
	# back-compat, if an adapter does not declare `supported_storage`
	# assume it supports file-based storage only.
	supported_storage = getattr(
	adapter, "supported_storage", {FileStorage}
	)
	for storage in self.context.writable_storage:
	if isinstance(storage, tuple(supported_storage)):
	break
	else:
	raise RuntimeError(
	f"The adapter {adapter} supports storage types "
	f"{[cls.__name__ for cls in supported_storage]} "
	"but the only available storage types "
	f"are {self.context.writable_storage}."
	)
	data_source = await ensure_awaitable(
	adapter.init_storage,
	storage,
	data_source,
	self.segments + [key],
	)

So, the task here is to check isinstance(storage, BucketStorage) versus FileStorage. Additionally, the supported_storage attribute on this class should be extended to include BucketStorage. This tells the caller to offer BucketStorage if that is the highest-priority item in writable_storage.

[danielballan]

Take a look at what SQLAdapter does here. If this is bucket storage, we need to get the credentials for this bucket, similarly to how SQLAdapter does. The credentials are not in the database, but they are in the config, and we need to match the URL of the bucket store with one in the config.

[danielballan]

tiled/tiled/adapters/sql.py

Lines 71 to 74 in f3331ef

	storage = parse_storage(data_uri)
	if isinstance(storage, SQLStorage):
	# Obtain credentials
	data_uri = cast(SQLStorage, get_storage(data_uri)).authenticated_uri

[danielballan]

No change to this function. Now, unlike when we started this PR, path_from_uri is only ever called on local filesystem paths.

[danielballan]

There won't be a bucket scheme. Instead, this function must be extended to accept dict (a YAML object from config). While we're here, we might accept a dict for PostgreSQL as well as for Blob/Bucket.

We currently accept SQL creds like this only:

- postgresql://username:password@host:port/database

but we could additionally accept a more structured input:

- uri: postgresql://host:port/database
  username: username
  password: password

And buckets would of course be similar.