429 TOO MANY REQUESTS on Zenodo Sandbox

[Eiafuawn]

When developing with the Zenodo API (invenio-rdm) sandbox, I often get 429 TOO MANY REQUESTS errors.

The sandbox rate limit is only 133 requests per minute for guest users, which is very restrictive and makes development difficult. Browsers send an OPTIONS request automatically before certain API calls, such as file uploads due to CORS policy, and these OPTIONS requests count toward the limit even though they don’t count as productive calls.

With the new API system, a record can have up to 100 files, but uploading 100 files will always exceed the 133 requests/minute limit because each file upload requires multiple requests, including OPTIONS ones.

This makes it impossible to fully test batch uploads without hitting the rate limit immediately. Raising the sandbox limit, excluding OPTIONS requests from the rate calculation, or providing an alternative testing setup would make development and testing significantly easier.

[carlinmack]

This is expected behaviour as there is a global limit of 100/req per minute https://developers.zenodo.org/#rate-limiting

It seems reasonable that uploading 100 files takes 1 minute

[lpatiny]

The problem is that uploading 1 file takes 4 requests so practically you can only upload 24 files per minute if you only upload files and do nothing else.

You have 2 calls to the files API (only once for all the files):

• files - OPTIONS
• files - POST

Then for each file you need 4 calls to upload your file:

• content - OPTIONS
• content - PUT
• commit - OPTIONS
• commit - POST

Could you at least ignore the OPTIONS query.

We upload at EPFL one zip per chemical sample in order to be able to visualize them independently. Typically we try to upload between 50 and 80 samples (so 50 to 80 zip files). Each sample is a zip of around 10Mb.

Here is an example: https://sandbox.zenodo.org/records/300911 and the possibility to visualize each sample individually: https://fair.cheminfo.org/v1/sandbox.zenodo.org/300911

Those are the requests to upload 2 files:

[carlinmack]

If you upload with the legacy API, you can upload with one request via

curl --request POST \
  --url {{baseURL}}/api/deposit/depositions/{{record_id}}/files \
  --header 'authorization: Bearer {{access_token}}' \
  --header 'content-type: multipart/form-data' \
  --form file=@file.txt \
  --form name=file.txt

With the REST API, it does indeed take 3 steps

• init file upload

curl --request POST \
  --url {{baseURL}}/api/records/{{record_id}}/draft/files \
  --header 'accept: application/json' \
  --header 'authorization: Bearer {{access_token}}' \
  --header 'content-type: application/json' \
  --data '[
  {
    "key": "file.txt"
  }
]'

• upload file content

curl --request PUT \
  --url {{baseURL}}/api/records/{{record_id}}/draft/files/file.txt/content \
  --header 'accept: application/json' \
  --header 'authorization: Bearer {{access_token}}' \
  --header 'content-type: application/octet-stream' \
  --data '{{file_data}}'

• commit file upload

curl --request POST \
  --url {{baseURL}}/api/records/{{record_id}}/draft/files/file.txt/commit \
  --header 'accept: application/json' \
  --header 'authorization: Bearer {{access_token}}'

These are also detailed here https://developers.zenodo.org/?python#quickstart-upload

Therefore you would find it quicker to upload via the legacy API instead of the browser, if that is available to you