Concatenation operation is not idempotent, causing duplicate uploads on retried requests

[ollie-sutton]

Describe the bug
When a client sends a POST request with Upload-Concat: final;url1 url2 ..., the server creates a new upload, concatenates the partial uploads, and returns 201 Created. If something in the request chain kills the connection before the response reaches the client due to a long file concatenation time (e.g. a WAF timeout rule, proxy timeout, or network interruption), the client never receives the response and tusd never emits the finished events/hooks. On retry, the server creates a new final upload with a new random ID and performs the concatenation again, wasting storage and triggering duplicate processing via hooks.

To Reproduce

1. Upload two partial uploads (Upload-Concat: partial)
2. Send a POST with Upload-Concat: final;/files/a /files/b
3. Kill the connection before the 201 Created response is delivered to the client (e.g. via a WAF timeout rule or proxy timeout)
4. Retry the same POST with Upload-Concat: final;/files/a /files/b
5. Observe that a second, duplicate final upload is created and concatenation runs again

Expected behavior
The concat operation should be idempotent. When the same set of partial uploads is presented for concatenation a second time, the server should detect that a final upload already exists for those partials and:

• If concatenation already completed (offset == size): return the existing final upload
• If the upload was created but concatenation didn't start (offset == 0): retry the concatenation on the existing upload
• Not create a duplicate final upload

Setup details

• Used tusd version: v2.9.2
• Used tusd data storage: Disk storage using a PVC within K8s
• Used tus client library: node tus client

[Acconut]

Thank you for bringing this up. This problem isn't specific to concatenation as no upload creation requests are currently idempotent. It's more likely to cause issues for concatenation requests where the processing times are higher and data volume is larger. But we've also seen this happening for normal upload creation requests where the client retries the requests and the server accumulates unused upload resources.

Tusd currently doesn't have any method to make upload creation requests idempotent or to identify retries. This mostly stems from the fact that the tus protocol itself doesn't address this problem at the moment.

An ideal solution is flexible enough to address the problem for any upload creation requests, not just concatenation requests. In the past, I've been thinking about adding support for the Idempotency-Key header field: https://www.ietf.org/archive/id/draft-ietf-httpapi-idempotency-key-header-07.html. Clients can set it and tusd will use it to detect retries and then avoid duplicate upload creations/concatenations. What do you think about this approach?

Regardless of the solution, we have to be careful with the state left behind by the first concatenation request. If the request is interrupted by a timeout, the upload may be left in an unexpected state and we need to ensure that tusd can continue nevertheless.

[m7kvqbe1]

We're hitting the same issue: large file concatenations timing out behind a WAF before the 201 Created response reaches the client, resulting in duplicate final uploads and duplicate hook processing.

One alternative approach would be an Idempotency-Key header, but I think the approach in #1365 is actually currently more resilient. The deterministic ID derived from the partial upload IDs is inherently durable, it survives server restarts without any additional state management. An Idempotency-Key header would require tusd to maintain a mapping of seen keys to upload IDs, likely in an in-memory set, which would be lost on restart.

Happy with either approach, but wanted to flag that the inferred composite ID in #1365 has this durability advantage out of the box currently.

[m7kvqbe1]

@Acconut

Please could you take a look at this implementation here, hopefully this covers all of your suggestions. It has durability when using local disk but not remote storage backends. That might not be perfect but I'd argue it's still a good improvement over the current situation?

#1366

[m7kvqbe1]

Tagging @Murderlon in this - we'd love to get this enhancement in if possible.

We'd be happy to build our own binary and run that ourselves for a while in our lower environments if you like to battle-test it.