Expose TUS capabilities through capabilities API

[PVince81]

Additionally to the headers (Tus-Version, Tus-Resumable, Tus-Extension) that are specified in a response of a PROPFIND or HEAD on a folder, we should also expose the general support for TUS through the capabilities API, so that a client does not need to bother with TUS related logic if the server doesn't have TUS enabled at all.

@michaelstingl @butonic @felix-schwarz @TheOneRing

[PVince81]

Proposal for JSON structure, under

"capabilities": {
   "core": {
      ...
   },        
   "files": {
      "tusSupport": {
         "version": "1.0.0,0.4.2",
         "resumable": "1.0.0",
         "extension": "creation,creation-with-upload"           
      }
   },
   ...
}

The capability key name is the same as the header, stripped of "Tus-" and lowercased.

if we decide to also implement the "max chunk size" header as per tus/tus-resumable-upload-protocol#93, we should also include those in the capabilities.

[michaelstingl]

if we decide to also implement the "max chunk size" header as per tus/tus-resumable-upload-protocol#93, we should also include those in the capabilities.

Yeah, needed for the middleware/proxy scenarios, and clients could immediately go for creation-with-upload.

[PVince81]

note that the recommended chunk size could also be different depending on the storage.
so if we put it in the capabilities it would be the "default chunk size" in case a storage doesn't specify any.

[michaelstingl]

if we put it in the capabilities it would be the "default chunk size" in case a storage doesn't specify any.

yes, exactly

[felix-schwarz]

Talking about middleware and proxies… this would also be a good place to instruct clients to send X-HTTP-Method-Override headers (tus.io) - and which HTTP method to use for the actual HTTP request instead.

For example:

"capabilities": {
   "core": {
      ...
   },        
   "files": {
      "tusSupport": {
         "version": "1.0.0,0.4.2",
         "resumable": "1.0.0",
         "extension": "creation,creation-with-upload",
         "http-method-override": "POST"            
      }
   },
   ...
}

[michaelstingl]

Talking about middleware and proxies…

@felix-schwarz I mean real size limits, that clients can't detect. Independent which method. Nothing can be sent > 100 MB. See my comment in owncloud/product#19 (comment).

Nasty middleware example:
PUT > 100MB will fail:
https://demo.shniq.cloud/
Cloudflare limits upload size (HTTP POST request size) per plan type:
100MB Free and Pro
(Source: https://support.cloudflare.com/hc/en-us/articles/200172516#h_51422705-42d0-450d-8eb1-5321dcadb5bc)

Or check traefik example:
https://docs.traefik.io/middlewares/buffering/

Clients need to know max-size before 1st upload.

[felix-schwarz]

@michaelstingl I wasn't referring to size limits. Sorry if my comment lacked clarity and/or context here.

What I meant is that http-method-override would - independently of any other capabilities describing other connection constraints - be good to have in here, too, in case f.ex. PATCH or OPTIONS are blocked by a proxy in front of the ocis server. Tus allows to add a header in that case, so that

OPTIONS /endpoint/ HTTP/1.1
Tus-Version: …
…

can also be sent as

POST /endpoint/ HTTP/1.1
Tus-Version: …
X-HTTP-Method-Override: OPTIONS
…

in case OPTIONS requests wouldn't get through. And via http-method-override, it would be possible to tell the client to use another HTTP method and provide the actual method via X-HTTP-Method-Override header. Like so:

…
"http-method-override": "POST" `
…

[PVince81]

• Reva adjustments: Add TUS global capabilities cs3org/reva#775
• ocis-reva adjustments: Make global TUS capability configurable #228
• ocis-reva update: TODO

[PVince81]

with the two above PRs, and using the go.mod approach, you can start the reva services after setting the env variables:

export REVA_FRONTEND_UPLOAD_MAX_CHUNK_SIZE=$(( 10 * 1024 * 1024 ))

it will make it advertise the given max chunk size in the capabilities

[michaelstingl]

yeah, builds nicely 👍

brew install hub
brew install go

git clone https://github.com/owncloud/ocis.git
git clone https://github.com/owncloud/ocis-reva.git
git clone https://github.com/cs3org/reva.git

cd ocis-reva
hub pr checkout 228
cd ..

cd reva
hub pr checkout 775
cd ..

cd ocis
echo 'replace github.com/owncloud/ocis-reva => ../ocis-reva' >> go.mod
echo 'replace github.com/cs3org/reva => ../reva' >> go.mod

make clean generate build

./bin/ocis server

Capabilities

% curl -u einstein:relativity -k 'https://localhost:9200/ocs/v1.php/cloud/capabilities?format=json' | jq '.ocs.data.capabilities.files , .ocs.data.capabilities.dav' 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1496  100  1496    0     0  42742      0 --:--:-- --:--:-- --:--:-- 42742
{
  "privateLinks": false,
  "bigfilechunking": false,
  "undelete": true,
  "versioning": true,
  "blacklisted_files": [],
  "tus_support": {
    "version": "1.0.0",
    "resumable": "1.0.0",
    "extension": "creation,creation-with-upload",
    "max_chunk_size": 0,
    "http_method_override": ""
  }
}
{
  "chunking": "",
  "trashbin": "1.0",
  "reports": [
    "search-files"
  ],
  "chunkingParallelUploadDisabled": false
}

@PVince81 is this correct? "max_chunk_size": 0, Default should be unlimited chunk size?

[michaelstingl]

export REVA_FRONTEND_UPLOAD_MAX_CHUNK_SIZE=$(( 10 * 1024 * 1024 ))
./bin/ocis server

With environment variable set, it looks good:

% curl -u einstein:relativity -k 'https://localhost:9200/ocs/v1.php/cloud/capabilities?format=json' | jq '.ocs.data.capabilities.files , .ocs.data.capabilities.dav'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1503  100  1503    0     0  26839      0 --:--:-- --:--:-- --:--:-- 26839
{
  "privateLinks": false,
  "bigfilechunking": false,
  "undelete": true,
  "versioning": true,
  "blacklisted_files": [],
  "tus_support": {
    "version": "1.0.0",
    "resumable": "1.0.0",
    "extension": "creation,creation-with-upload",
    "max_chunk_size": 10485760,
    "http_method_override": ""
  }
}
{
  "chunking": "",
  "trashbin": "1.0",
  "reports": [
    "search-files"
  ],
  "chunkingParallelUploadDisabled": false
}

Didn't test with remote setup, but don't expect trouble:
https://owncloud.github.io/ocis/basic-remote-setup/#use-the-binary