[Feature Request] Support creating operator from resource URI

[frostming]

Proposal

I propose an alternative approach to create an opendal Operator -- from a resource URI(or URL? either is good) string.

I am new to OpenDAL and feel free to close it if it is already a thing -- That's great!

Examples:

let op = Operator::from_uri("file:///tmp")?.finish();

This does the same as

let mut builder = Fs::default();
builder.root("/tmp");
let op: Operator = Operator::create(builder)?.finish();

Another example, HDFS:

let op = Operator::from_uri("hdfs://127.0.0.1:9000/tmp")?.finish();

Advantages

• Of course, this can save us a few lines of initialization code.
• This will also make the configuration easier -- only one config line is needed.
• Language bindings can choose to not expose Builder structs at all. For a minimal interface, an Operator struct is sufficient, service backends can be set up with a primitive string. This will significantly save the effort of supporting a new language.

Possible Solutions

We already have a unique Scheme for each Builder, we can give them a unique URL prefix. The URI will be encoded as follows:

$scheme://$positional_arg1/$positional_arg2?arg1=value1&arg2=value2

Where positional arguments are required parameters for the specific service, such as root for Fs. Specifically, the URI will be parsed with Url::parse, and the query_pairs will be fed to the ::from_iter() method of the corresponding Builder to get a builder object. Thanks to the good API design of OpenDAL, these APIs are ready to do this.

Be aware because all values are URL components, they must be percent-encoded.

Some concerns

For some services that use URL as the parameter themselves, such as IPFS, the resource URI will be a bit different:

ipfs://...... # for http endpoint
ipfss://...... # for https endpoint

And there can be multiple schemes map to the same builder. Both http:// and https:// map to HTTP builder.

This might cause some confusion, so I will leave it to the maintainers to make the choice of how this feature will go.

[Xuanwo]

A great proposal to address #1424 👍

For some services that use URL as the parameter themselves, such as IPFS, the resource URI will be a bit different:

This could be trick for services like s3, oss:

s3s://example.com/bucket
osss://example.com/bucket

Most HTTP-based services require an endpoint that includes a scheme such as https://example.com or http://example.com. However, I am unsure if this is a suitable solution:

s3://bucket/path/to/root?endpoint="https://example.com"

This issue has led me to a new design, like the following:

s3:bucket="bucket",endpoint="https://example.com",...

[suyanhanx]

Looks really like a database or some sort of resource connection string.

[flaneur2020]

s3://bucket/path/to/root?endpoint="https://example.com"

I guess this manner is ok, as URI just accepts parameters, endpoint is just an parameter.

s3:bucket="bucket",endpoint="https://example.com",...

The format may not be as easy as it seems since we have to deal with things like escape characters, Unicode, and more. In such cases, parsing a URI using standard parsing semantics is much simpler

[frostming]

However, I am unsure if this is a suitable solution:

s3://bucket/path/to/root?endpoint="https://example.com"

Theoretically any string can be encoded in the query string using URL encoding, so the example should be:

s3://bucket/path/to/root?endpoint=https%3A//example.com

But yes, the value must be a string, if they are of other types, we need type conversion.

[Xuanwo]

Forwared from #1905 (reply in thread)

I was wondering if there is a direct way to create an operator by simply providing a URL and backend service-specific options.

This idea is also implemented by,

PyArrow FileSystem API fs.FileSystem.from_uri(uri)
Hadoop FileSystem API org.apache.hadoop.fs.FileSystem.get(uri, conf)
Fsspec fsspec.filesystem(protocol, **storage_options)

I have created the following on object_store side with more details.
https://github.com/apache/arrow-rs/issues/4047

[chitralverma]

@Xuanwo do you want to do this or shall I take a stab at this ? :)

[Xuanwo]

Please correct me if I understand wrongly.

---

We already have Operator::from_map which can be used like:

let options: HashMap<String, String>;

let op: Operator = Operator::from_map::<Fs>(map)?.finish();

And now we are trying to add something like:

let options: HashMap<String, String>;

let op: Operator = Operator::from_uri("s3://<bucket>/<path>", options)?.finish();

[suyanhanx]

It may merge options into uri.

[chitralverma]

let op: Operator = Operator::from_uri("s3://<bucket>/<path>", options)?.finish();

Yes this is what I mean, but it will be better to call it from_url considering HTTP and all

[chitralverma]

FYI, something similar I am trying to do for object_store

apache/arrow-rs#4077

[chitralverma]

@Xuanwo @frostming , currently I'm using something like this over a typical Url as a naive hack. I believe something more sophisticated can be added on similar lines?

FYI, link to something similar that object_store does - link

    pub fn infer_operator(&self, opts: HashMap<String, String>) -> Result<Operator, Error> {
        let mut _opts = opts;

        let scheme = match (self.url.scheme(), self.url.host_str()) {
            ("file", None) => Scheme::Fs,
            ("az", _) => Scheme::Azblob,
            ("adl" | "adls" | "abfs" | "abfss" | "azdfs" | "azdls", _) => Scheme::Azdfs,
            ("s3" | "s3a", Some(bucket)) => {
                _opts.insert("bucket".to_string(), bucket.to_string());
                Scheme::S3
            }
            ("gs", Some(bucket)) => {
                _opts.insert("bucket".to_string(), bucket.to_string());
                Scheme::Gcs
            }
            (scheme, _) => {
                return Err(OpenDalError::new(
                    OpenDalErrorKind::Unsupported,
                    format!("Unable to recognise URL with scheme \"{scheme}\"",).as_str(),
                )
                .into());
            } 
        };

[Xuanwo]

Sounds great. I'll prepare a proposal following the v0.40 release to initiate discussion on this. Thanks so much!

[flaneur2020]

are there some further progresses after this discussion?

[chitralverma]

@Xuanwo any updates on this?

[Xuanwo]

@jorgehermo9 is currently working on this now: #5482

I'll get back to reviewing it once I've finished some urgent $$$ work.

[Kinrany]

For S3 and other HTTP-based centralized services it may make sense to use https: and the domain name.

[Kinrany]

On the other hand, S3 specifically could also be interpreted as an unofficial protocol, seeing how many other implementation aim to be compatible.