[Issue] Gateway with embedded network requester crashes at `common/nymsphinx/chunking/src/reconstruction.rs:108:26`

[numbleroot]

Describe the issue

Setup and procedure are the same as for #4406, thus I've copied and modified the report text below.

On nym-binaries-v2023.5-rolo, when running a gateway with embedded network requester (initialized by adding flags --with-network-requester --open-proxy true) with at least 4 endpoints constantly sending HTTP requests towards this embedded network requester, the gateway sometimes crashes with the following panic:

thread 'tokio-runtime-worker' panicked at common/nymsphinx/chunking/src/reconstruction.rs:108:26:
index out of bounds: the len is X but the index is Y

with Y >= X, as these values differ across the instances I've observed.

I've seen this panic crash a gateway on at least 3 individual attempts at running my endpoints-to-embedded-network-requester setup. It can take quite a while to occur, but typically happens eventually.

Expected behaviour

The gateway should not panic.

Stack traces

thread 'tokio-runtime-worker' panicked at common/nymsphinx/chunking/src/reconstruction.rs:108:26:
index out of bounds: the len is X but the index is Y

with Y >= X, as these values differ.

Steps to reproduce

Initialize and run a Nym nym-binaries-v2023.5-rolo gateway with embedded network requester configured to act as an open proxy. Have 4 or more HTTP clients hooked up to their own independent nym-socks5-clients send constant HTTP requests (e.g., by iterating through a list of URLs) to this embedded network requester through Nym. I've set up all involved Nym endpoints to be registered with this one gateway (i.e., the embedded network requester and all nym-socks5-clients use this one gateway as their gateway, no other gateway involved). Wait for the panic to occur eventually, maybe after some panics of type #4406 (it seems the "index out of bounds" issue occurs less frequently than the "Tried to insert duplicate pending ack" issue). As a side note, I'm typically seeing gateway logs indicating around 300 concurrently ongoing proxies at the gateway when this error occurs.

Screenshots

n/a

Which area of Nym were you using?

• Application: nym-gateway (with embedded network-requester), nym-socks5-client
• OS: Ubuntu 22.04
• Version: nym-binaries-v2023.5-rolo

Additional context

I currently have no theory for why this error occurs.

[octol]

This looks like a bug, we need to fix this

[octol]

Self-assigning this to myself so we don't forget about it