Socket stalls sending big files

[kyordhel]

Hello there. I'm having similar issues as described by QuirkyCort and OscamSatUser somewhere else regarding speed while sending rather "large" chunks of data. In my case, the code just hangs and the device requires a reset, even with a timeout. I need to transfer a ~30kB file.

So far socket.sendall(file.read()) yields the best results but still hangs when transferring the largest file at random. I've tried using both blocking and non-blocking writes (socket.write) with a 100ms timeout, as well as reading the file in chunks of 256, 512, 1024, 2k and 4k bytes. I've tried write, sendall, and even handling the socket with makefile. All approaches hang.

Using MicroPython v1.26.1 on 2025-09-11; Generic ESP32 module with ESP32. For context, all this runs in a separate thread started with start_new_thread(_webserver_task, []). I understand _thread is experimental and not really supported but these are the lemons I chose for my lemonade. There is another separate thread also accepting connections on a different port that reads no file that works flawlessly but never transfers or receives data over 512 bytes.

More or less code goes like this.

def _webserver_task():
    ...
    _sckt.listen()
    while True:
        cnn, addr = _sckt.accept()
        request = cnn.recv(1024).decode('utf-8')
        _serve_file(cnn, request)
        gc.collect()

Where:

def _serve_file(cnn, request):
    file = _get_file_from_request(request)
    with open('/web/public_html' + file, 'rb') as f:
        cnn.sendall(f.read())

[shariltumin]

Have you try to read and send the big file in chunks, say 512 bytes in size?

with open(file_path, 'rb') as f:
    while True:
        chunk = f.read(512)
        if not chunk: break
        cnn.sendall(chunk)

[kyordhel]

Yes, I did. Chunks of 256, 512, 1024, 2k and 4k bytes as I stated in the second paragraph p[214:286].
I tried also read chunks to a pre-allocated bytearray of variable size (again 256, 512, 1024, 2k and 4k bytes) using file.readinto(buff). and then send the chunks or buffer with sendall, write, and send.

The first n chunks are sent correctly, but at some point the execution stalls.

I even tried this:

i = 0
sent = 0
cnn.settimeout(100) #cnn is a socket
with open('/web/public_html' + file, 'rb') as f:
    chunk = f.read(1024)
    while chunk:
        i+= 1
        count = cnn.write(chunk)
        sent+= count
        # print(f'Sent chunk {i} of {file}, {count}bytes written for a total of {sent}bytes sent')
        f.seek(sent)
        chunk = f.read(1024)

[davefes]

For the ESP32 SPIRAM variant I had to:

      # send data in 1024 chunks
        with open('datalog.csv', 'rb') as file:
            for chunk in range(0, file_size, chunk_size):
                this_chunk_size = min(chunk_size, file_size-chunk)

              # handles the last dangling chunk that would be smaller than the full 1024 bytes
                smtp.write(file.read(this_chunk_size))
                time.sleep_ms(200)  # guarantee consecutive writes

the sleep sorted it. Maybe, it might help on the ESP32 Generic?

[kyordhel]

This looks interesting, @davefes. The sleep_ms could maybe be skipped if you used sendall instead?

Unfortunately didn't work either. It still stalled after serving 25 chunks of 1kb (stall, no crash) on the third attempt (the first two didn't make it past the 9th chunk).

Again, sending seems to fail whilst in a thread and reading files.

[davefes]

Didn't realise it was threaded, good luck! I didn't read it closely enough. What about asyncio?

[shariltumin]

I assume that your client and server are connecting via HTTP.

If you are sending data (text) from a file in chunks over HTTP, you need to provide proper HTTP headers in your reply.

    # a part of _serve_file function

    # --- Send headers ---
    cnn.write(b"HTTP/1.1 200 OK\r\n")
    cnn.write(b"Content-Type: text/html\r\n")
    cnn.write(b"Transfer-Encoding: chunked\r\n")
    cnn.write(b"Connection: close\r\n")
    cnn.write(b"\r\n")

    # --- Send file in chunks ---
    while True:
        chunk = f.read(CHUNK_SIZE)
        if not chunk:
            break
        cnn.write(b"%x\r\n" % len(chunk))
        cnn.write(chunk + b"\r\n")
        time.sleep(0.01)

    # --- End of chunks ---
    cnn.write(b"0\r\n\r\n")

    # close to release resources
    f.close()
    cnn.close()

[shariltumin]

I've run out of ideas and still don't know why your threaded web server isn't working.

In an attempt to duplicate the problem you were experiencing, I wrote a simple threaded web server. If you're interested, take a look at:

https://github.com/shariltumin/bit-and-pieces/tree/main/threaded_web_server

I have no problem running a two-port web server using threads. In fact, I use separate threads each time to service client requests using curl.

[kyordhel]

Amazing! Thank you very much, @shariltumin will definitively peek a look.

[kyordhel]

Indeed. Your snippet works. I do not understand the purpose of those delays there, tho. I'm stealing the flags for the listener..

I copied your file_uploader function quasi-verbatim. Only changes were the filename with a variable to serve html, css, and image files as needed; and commenting the content-type from the header. It stalled again. Sometimes while serving index, most of the times while serving the CSS or PNG files. In this regard, your function behaves no different from mine and providing a header makes no difference.

Guess I'll have to drop the web server entirely until I get time to port this to C.

[shariltumin]

The delay function works similarly to await asyncio.sleep(delay). It pauses execution on the current thread and hands over CPU and other system resources to the underlying RTOS.

Consider this:

# test_sleep.py
from time import sleep_ms, ticks_ms, ticks_diff
import _thread as th

def test(n,w):
    sleep_ms(w)
    print(f'{n} exit after {w} ms')

t0 = ticks_ms()
th.start_new_thread(test, ('ONE', 400))
th.start_new_thread(test, ('TW0', 200))
th.start_new_thread(test, ('THREE', 100))

sleep_ms(400+4) # make sure main timeout last
t1 = ticks_ms()
print(f'Main task exit after {ticks_diff(t1,t0)}ms')

Running this script will take 404ms+, not the sum of these delays.

$ mpr u0 mount . run test_sleep.py
Local directory . is mounted at /remote
THREE exit after 100 ms
TW0 exit after 200 ms
ONE exit after 400 ms
Main task exit after 404ms

The srv.listen(5) accepts a maximum of five concurrent client connections. If you receive many warnings, it means your program is allocating client sockets as many times as the number of warnings by cs, ca = srv.accept(). If these sockets are not closed properly, you will run out of socket resources very quickly. That may be the root cause of the problem.

If you want to use standard tools like cURL or wget to test your server, you need proper HTTP headers. If you don't need HTTP, you can use a raw TCP socket for both the server and clients.

Making a C program work is much harder than a MicroPython program. You might want to try asyncio before attempting C.

Whatever you decide, I wish you success!

[kyordhel]

Thanks again. I am more familiar with how FreeRTOS works than with how MicroPython is implemented, so your information is quite valuable.

All this trouble is to load some images (now svgs) so we can see what the robot "sees" in a browser, so curl/wget are pretty much useless to us. My tcp raw sockets seem to work perfectly, whilst only the ones that send files to the browser stall. Sending json data crafted from sensors is not an issue and all web connections are closed at the end. Still no clue what I am doing wrong.

Ultimately, this python firmware is just palliative care whilst the C solution gets coded, so rather than invest time in finding the bug I should just move on to the C part.

Thanks again for all the advise. I learned a lot.