Consider using a standardized API for file access

[jdries]

We have currently defined our own API for sharing files with OpenEO.
The S3 API is also a well known http-based file api (object storage).
I'm not an expert, so this is really more like a question to investigate if this would be usable.
If S3 covers all of our requirements, using it would simplify our own API, and also backend impementations as it is very widely adopted and supported by existing software.

[m-mohr]

I'd really like to adopt a well-known/well-defined API for file management. I'm also not an expert in S3 or other potential file-related HTTP APIs. Anybody here has experience? A first look at the S3 REST API makes me feel that it is a bit too complex for our "simple" implementations. I'm not sure yet whether that API could be stripped down to just allow a minimal subset, which I'd say is mandatory to keep the openEO API simple in this regard. If it needs to be fully implemented I could only see that to be added as an extension. Are there other file APIs we could adopt? I found Azure and Google of course.

[edzer]

Ideally this would be back-end heterogeneity that we would abstract away in the openEO API.

[m-mohr]

@edzer That is what we are trying at the moment with our file API, but it is a bit limited and proprietary. If there is a standard that could be adopted with a good ecosystem, it would be a good idea to adopt it. Not sure whether the existing cloud services as S3, Azure, GCS could handle that as they usually have service specific things in their APIs. So we are basically looking for an existing standard that already did the abstraction. If there is none, we probably continue with what we have at the moment.

[edzer]

GDAL supports the /vsi prefixes: /vsizip/, /vsis3/, /vsigcs/ etc see here that abstracts over many cases operationally, i.e. it is a working implementation. It does mean that a script needs to be adapted when porting from AWS to GCS.

[edzer]

But that might be OK (and could even be automated).

[m-mohr]

@edzer As discussed, that could be useful for back-end implementations, but I don't see a direct benefit for the API specification. I'm more looking for something like a simple and "modern" WebDAV.

[m-mohr]

Maybe remoteStorage is what we are looking for: https://remotestorage.io/
Some thoughts: https://unterwaditzer.net/2015/kill-webdav.html

The only thing that is more complex in remoteStorage than in WebDAV is authentication. RemoteStorage requires the server to support a subset of OAuth, and that's the only kind of authentication supported. It also requires WebFinger support instead of making it optional (like in WebDAV, where it's almost a luxury if the DAV client actually finds the HTTP endpoints it's supposed to use).

Sound great, but I'm wondering how we can integrate that given the fact that we need to merge the openEO and remoteStorage authentication procedures somehow.

Another interesting repo to look at is https://github.com/scality/cloudserver

[mkadunc]

Maybe remoteStorage is what we are looking for: https://remotestorage.io/

I also like remoteStorage a lot, but it has a long way to go before it replaces S3 API as the go-to REST interface.

The industry seems to have settled on S3's interface for object storage - in addition to scality/cloudserver , many other solutions use the same API or provide S3-compatible proxy to GCS and others, e.g.
Min.io , Ceph, OpenStack Swift.