KeepAliveDisconnected from big files or long upload chunk sessions

[phgrandin-venturus]

Hi team, I’m trying to investigate the following scenario:

I’m having an issue with the chunk files with more than 2GB-6GB. It seems there is something that is not letting the session opened for a long time. After around 2 hours of uploading the chunked file parts in a session, it unexpectedly stops the process and closes the session: DELETE /api/2.0/files/upload_sessions/:sessionID.

Usually this refreshes the token as well, and then finally closes the session.
The final result and log for us shows a KeepAliveDisconnected error from HTTPClient:

..../bundle/ruby/3.1.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:524:in `rescue in query': HTTPClient::KeepAliveDisconnected: Broken pipe (HTTPClient::KeepAliveDisconnected)
/usr/local/lib/ruby/3.1.0/openssl/buffering.rb:354:in `syswrite': Broken pipe (Errno::EPIPE)
..../bundle/ruby/3.1.0/gems/httpclient-2.8.3/lib/httpclient/session.rb:524:in `rescue in query': HTTPClient::KeepAliveDisconnected: Broken pipe (HTTPClient::KeepAliveDisconnected)
/usr/local/lib/ruby/3.1.0/openssl/buffering.rb:354:in `syswrite': Broken pipe (Errno::EPIPE)

I tried many timeout increases from my side, including from the HTTPClient, but none of them worked:

Boxr::BOX_CLIENT.debug_dev = Rails.logger

Boxr::BOX_CLIENT.send_timeout = 3600 * 24
Boxr::BOX_CLIENT.receive_timeout = 3600 * 24
Boxr::BOX_CLIENT.keep_alive_timeout = 3600 * 24
Boxr::BOX_CLIENT.connect_timeout = 3600 * 24
Boxr::BOX_CLIENT.protocol_retry_count = 25

I've tried also using the threads implementation, to send parallel requests to upload in a shorter period of the session. But it seems more things goes wrong, and the final result is similar:

BoxService.new.chunked_upload_file('/tmp/arch_6gb.zip', '<FOLDER_ID>', name: 'arch6.zip', n_threads: 10)

I appreciate if anyone could help me finding any clue about this. Thanks!

[xhocquet]

Hey there, apologies for the late reply 😄 Is this still something you actively experience?

When you create the upload sessions, what session_expires_at value do you get back?

Based on that error message, it would mean something on the other end of the server is terminating the connection. That could be Box, or any of the network layers.

One thing I'm curious about is the keep_alive_timeout. Maybe Box isn't happy with the long-lived connections, assuming the same connection is held throughout the whole 2 hours? That's certainly longer than what I allow on webservers I've worked on.

Maybe you could try Boxr::BOX_CLIENT.keep_alive_timeout = 0?

Also if you have a file that isn't to sensitive that can reproduce the issue, that would be helpful. I have a larger file as part of smoke tests, but it succeeds 100% of the time.