How to resume an upload with updated custom headers (e.g. refreshed auth signature)

[deJong-IT]

We're using TUSKit to upload files to Bunny.net.
Each upload requires a signed authorization signature, so we start uploads using:

tusClient.uploadFileAt(filePath: url, customHeaders: headers)

When the upload is interrupted or paused (for example, when the app is closed or internet is lost), we want to resume the upload later.
However, by that time our authorization signature may have expired.

Is there a way to resume an existing upload with updated custom headers (e.g. a new Authorization token/signature)?
If not, would it be possible to expose a way to modify headers before retrying or resuming?

Thanks for the great library!

[alex-gillespie]

I am running into this same issue. Trying to figure out what the best way to update the auth token in a resumed uploads header is.

[donnywals]

Hi Folks! Apologies for the silence on this issue...

We added something like this to TransloaditKit a little while back: https://github.com/transloadit/TransloaditKit/pull/43/files

I'm wondering whether this pattern would work for you. You'd be able to plug a signature generator into TUSKit (either at init or per-upload. I'm not sure yet) and this would allow us to ask you for a signature every time we're about to start an upload, regardless of whether we're resuming, retrying, or just starting.

I think it would make sense for us to pass the headers we'd normally use back to the client, you'd be able to modify or augment them, and we'd use the returned headers. So when you would be able to use the same signature, you just return the unmodified dictionary. If you do need to change something, you return the changed dictionary.

I'm aiming to get this into a PR within the next week or so. Again, apologies for leaving this one unanswered for a little while...

[jjnguy]

No apologies needed! Thanks for looking into this. We're working on a solution that just recreates the whole client in order to refresh credentials. But we agree that a dynamic request header system would be a better solution.

[donnywals]

I'm currently looking at a solution that will do the following:

• for every request, call a closure that you provide to TUSClient
• the closure will receive a dictionary of custom headers that were provided to the original request along with information about the request (so you can distinguish which request you're working on)
• the closure also receives a handler that you can call with your modified header dictionary. This allows you to asynchronously build your modified headers (if needed)
• we then merge your custom headers with our headers and send the request

[alex-gillespie]

Hi Folks! Apologies for the silence on this issue...

We added something like this to TransloaditKit a little while back: https://github.com/transloadit/TransloaditKit/pull/43/files

I'm wondering whether this pattern would work for you. You'd be able to plug a signature generator into TUSKit (either at init or per-upload. I'm not sure yet) and this would allow us to ask you for a signature every time we're about to start an upload, regardless of whether we're resuming, retrying, or just starting.

I think it would make sense for us to pass the headers we'd normally use back to the client, you'd be able to modify or augment them, and we'd use the returned headers. So when you would be able to use the same signature, you just return the unmodified dictionary. If you do need to change something, you return the changed dictionary.

I'm aiming to get this into a PR within the next week or so. Again, apologies for leaving this one unanswered for a little while...

That looks like it could work. I think just provided at client creation would be enough. I'm not sure if I can think of a reason that different uploads from the same tus client going to the same destination would have a reason to need different signature generators.

[donnywals]

I'm not sure if I can think of a reason that different uploads from the same tus client going to the same destination would have a reason to need different signature generators.

Agreed, I think providing the closure on the client level should provide more than enough flexibility. Especially if you could theoretically still access the upload identifier if needed so you could send some kind of custom information for specific uploads.

[alex-gillespie]

The stop gap solution we were working on (wiping out the client and rebuilding it with a current token whenever we get an auth error) appears to be untenable due to the scheduler continuing to schedule task in the background after stopAndCancelAll is called and having no practical way (That I can tell by looking through the source code, I may have missed something I guess) to wait for the scheduler to be done scheduling task. And since setting the TusClient reference to null before the Scheduler is done seems to invalidate the underlying URLSession, it causes a crash due to an unhandled exception. It may be worth adding a method to the TusClient class that allows a consumer to wait for the schedule to finish all things its doing for something like this. I also looked through the code you have pushed up on your working branch and it also looks like it would solve our problem so maybe it wouldn't be needed.

[donnywals]

I've done a bit of testing on this branch and to me it looks like this should get us the functionality that's being requested.

@alex-gillespie @jjnguy if you're able to, would you be willing to give this branch a try and let me know whether this suits your needs? #218

[alex-gillespie]

Yeah, I'll try using it today and let you know how it goes.

[donnywals]

Thanks Alex! Always useful to get some feedback on features like these before releasing them.

[alex-gillespie]

@donnywals Maybe I've set something up wrong but the header generator never seems to get called. Are there specific conditions that have to be met before it gets called? This is my tus client set up. I never see my print statement in the console and break points inside don't hit

` let config = URLSessionConfiguration.background(withIdentifier: "com.identifier.tus")
config.httpAdditionalHeaders = [:]
config.httpAdditionalHeaders?["Authorization"] = "Bearer (accessToken)"
config.httpAdditionalHeaders?["User-Agent"] = "WestShoreHome.identifier.Swift"
config.httpAdditionalHeaders?["WSH-Auth-Provider"] = "Azure"

    let server = URL(string: ApiClient.url)!
    
    // Use a concrete file URL in Caches directory for TUS storage (thread-safe, app-writable)
    let caches = FileManager.default.urls(for: .cachesDirectory, in: .userDomainMask).first!
    let storageDir = caches.appendingPathComponent("TUS", isDirectory: true)
    try? FileManager.default.createDirectory(at: storageDir, withIntermediateDirectories: true)
    
    tusClient = try? TUSClient(
        server: server,
        sessionIdentifier: "identifier",
        sessionConfiguration: config,
        storageDirectory: storageDir
    ) { [self] uploadID, headers, onHeadersGenerated in // Custom Header Generator
        print("\n\n\n----- Header Generator Called ----- \n\n\n")
        Task {
            guard let success = await refreshAccessToken?(), success else { onHeadersGenerated(headers); return }
            guard let newAccessToken = self.accessToken else { onHeadersGenerated(headers); return } // if there is a problem with refreshing the access token the tusClient should attempt to make the api call with the current headers

            var mutatedHeaders = headers
            mutatedHeaders["Authorization"] = "Bearer \(newAccessToken)"
            onHeadersGenerated(mutatedHeaders)
        }
    }`

[donnywals]

The generator should always be called for every upload you start. Just to make sure, how are you starting your upload?

[alex-gillespie]

Uploads are started with the normal uploadFile function
if let id = try? client.uploadFileAt(filePath: filePath, uploadURL: fullUrl, context: contextWithType)
the Tus Client Start function is called on app launch to resume any paused uploads along with a check for all failed uploads from potential background upload failures and a call to retry or removeCache is the retry fails.

[donnywals]

Ahh, right.. so I designed this so that the generator would only be called when you pass custom headers in the first place. When you don't pass them to the upload function, the generator isn't called.

I'm second guessing this now. It probably makes more sense to just call the generator regardless of custom headers being passed initially.

WDYT?

[alex-gillespie]

My initial assumption would tell me that a header generator specified in the initializer would run every time. If you think there should be a reason for it to not run on some uploads maybe a simple boolean passed into the uploadFile functions (defaulting to true? or maybe false?) that determines if the header generator would run for that uploads request would be better? Allowing the user of the TUSKit api to still have control over if the generator runs but also making the default functionality a little bit more of what a developer might expect at a glance?

I could see an argument for all 3 options though. In the end I'd probably lean towards always run or the boolean instead of only when there's custom headers passed into the upload call.