Context
I'm got a project where I return s3 input streams as :body, as a way to send s3 data through my app without loading the entire file in memory all at once. This works fine most of the time. But when the http request is aborted early, .close is not being called on the inputStream. In the case of java s3, not closing the inputSteams is quite bad as in that case, the library stops working after 50 open requests.
Example:
..
(:require [amazonica.aws.s3 :as s3])
...
(GET "/stream" _
{:status 200
:body (:input-stream (s3/get-object {:endpoint region} bucket key))})
curl http://localhost:4000/stream --compressed | head -c 5 > /dev/null
The above will not correctly close the s3 stream - I've verified this using a debugger.
Suspicious code
https://github.com/clj-commons/ring-gzip-middleware/blob/4328168b640ce5006efcb11b371e5f1c9250204e/src/ring/middleware/gzip.clj#L37C1-L53C14
(defn piped-gzipped-input-stream [in]
(let [pipe-in (piped-gzipped-input-stream*)
pipe-out (PipedOutputStream. pipe-in)]
; separate thread to prevent blocking deadlock
(future
(with-open [out (if @flushable-gzip?
(GZIPOutputStream. pipe-out true)
(GZIPOutputStream. pipe-out))]
(if (seq? in)
(doseq [string in]
(io/copy (str string) out) <-------- io/copy throws IOException when out is closed early
(.flush out))
(io/copy in out)))
(when (instance? Closeable in)
(.close ^Closeable in))) <--- in that case, .close isn't called! (should be in a finally)
pipe-in))
Thanks for your attention!
Context
I'm got a project where I return s3 input streams as :body, as a way to send s3 data through my app without loading the entire file in memory all at once. This works fine most of the time. But when the http request is aborted early, .close is not being called on the inputStream. In the case of java s3, not closing the inputSteams is quite bad as in that case, the library stops working after 50 open requests.
Example:
The above will not correctly close the s3 stream - I've verified this using a debugger.
Suspicious code
https://github.com/clj-commons/ring-gzip-middleware/blob/4328168b640ce5006efcb11b371e5f1c9250204e/src/ring/middleware/gzip.clj#L37C1-L53C14
Thanks for your attention!