Building chunks as they come in

[ariselseng]

Hi,
I am using concurrent chunks because of the speed benefits.
In my setup I have the final files folder on a mounted samba share which has a limited write speed.
The assembling of chunks can therefore take up to 15 minutes.

Would it be possible to somehow make the assembling happen as the chunks come in, and the that the final assemble just be a rename like "bigfile.mov.assembling" -> bigfile.mov and deleting of chunks?

Would you guys merge a pull request for this or do you see this as out of scope/not in fineuploaders api?

[rnicholus]

I'm not sure this is possible, since there is no guarantee that chunks will arrive/complete in order. This is especially true when the concurrent chunking feature is enabled client side.

[ariselseng]

@rnicholus
With a goroutine in the background that waits until the next needed chunk arrives it is certainly possible. When a chunk arrives, the server needs to communicate with that goroutine somehow and make it add it to the file. The last request to assemble the file could be a simple rename from a temporary filename to the real one.

[rnicholus]

This sounds...complicated. I'm also concerned about alerting the client about errors combining the files if this happens out-of-band.

[orlandovald]

Agree it sounds complicated, the problem, as Ray mentioned, is that the sequence can't be guaranteed when concurrent chunks are sent and they could get out of sync, you'll need to keep track of the next chunk in the sequence to be appended and leave on hold any chunks that arrive out of sequence, checking when to append the ones on hold vs others that could still be arriving, etc not impossible but complex. Do you have some other mount with faster IO and less latency that you can use as "staging area" to receive and merge the chunks, finally you'll just need to move the file to the samba share, not exactly what you are asking but might be faster as you will avoid the merging and chunk clean up on the slow mount.