fix(backend): spool released exports safely - #6926
Conversation
Build each export from one successful query result so X-Total-Records matches the served bytes and the database transaction ends before transfer. Write security headers before the async file copy. Bound spool storage and clean it across retries, failed handoffs, timeouts, disconnects, and restarts. Align the bundled download clients and deployment timeouts with generating the full spool before the first byte is sent. Refs #3511, #5198
clonder
added
bug
claude
Bot
added
backend
There was a problem hiding this comment.
@clonder and @corneliusroemer I heard you guys did some perf of this new feature - could you maybe summarize your results in the PR description? would be nice to have this for future reference :-)
| return when (compressionFormat) { | ||
| CompressionFormat.ZSTD -> ZstdCompressorOutputStream(checkedFileStream) | ||
| null -> checkedFileStream |
There was a problem hiding this comment.
As the data can be very large, I think that we should always compress them when writing to disk. If the user has requested compressed data, the compressed file can be directly returned. If the user does not want compressed data, they would be decompressed again when returning to the user.
There was a problem hiding this comment.
Thanks @clonder for the PR and @corneliusroemer for the proposal. Like Anya, I'd also be quite interested in seeing the performance impacts; maybe let's merge #6807 first and use it to compare the performances?
3 participants
materialize each released-data export from a single successful run of the released-record stream. count records while writing the temporary file so
X-Total-Recordsmatches the served bytes and the database transaction closes before network transfer begins.because the backend now prepares the complete export before sending the first byte, increased the read timeouts for the bundled SILO and CLI clients and configure bounded spool storage in the deployment.
Resolves #5198
Related to #3511
🚀 Preview: https://fix-get-released-data-sin.loculus.org