BIG BUG: filehandle leak on latest version of MinKnow, "script error" on sequencing, terminates after 18hrs

[GabeAl]

Ubuntu 24.04. After upgrading 2 weeks ago to the latest MinKnow version, we have been encountering a persistent error.

After ~18 hours of sequencing, highly multiplexed promethion runs (e.g. 2x96) runs fail with "Script error." The rapid kit shows this issue, but we have not tested with the native yet.

Digging into the error, I see the log files mention a very curious but telling error:

/var/log/minknow/mk_manager_svc_log-1.txt:298:                   std::terminate() called: Failed to create pod5_file: IOError: Failed to open local file '/var/lib/minknow/data/reads/tmp/P2S-03174-A/WP004_mk3/no_sample_id/20250805_1051_P2S-03174-A_PAY42344_f4ecd010/pod5/.5285fcae-479d-4408-b9b5-76351f0fc05b.tmp-reads'. Detail: [errno 24] Too many open files
/var/log/minknow/mk_manager_svc_log-1.txt:519:                   std::terminate() called: Failed to create pod5_file: IOError: Failed to open local file '/var/lib/minknow/data/reads/tmp/P2S-02548-A/WP005/WP005/20250807_1609_P2S-02548-A_PBE93237_fae0efa4/pod5/.1f2b1dd7-932a-4555-b854-c27531a01899.tmp-run-info'. Detail: [errno 24] Too many open files
/var/log/minknow/mk_manager_svc_log-2.txt:312:    detailed_error_info: Network transport error: Operation timed out after 60000 milliseconds with 0 bytes received: Timeout was reached

Indeed, monitoring the file handles open over the course of a run after forcibly removing the file descriptor limit on the services is damning:

Something is screwing up in this version of MinKnow where it doesn't close its files properly. For small, short, or low-multiplex runs this doesn't rear its head as much, but for long heavily multiplexed runs it's a showstopper.

Please let me know what help I can provide to get this fixed. My temporary solution has been to adjust the .service file to
[Service]
LimitNOFILE=524288

and in bash

# bump limits for all relevant ONT processes
for p in $(pgrep -f 'minknow|control_server|basecall_manager|dorado'); do
  sudo prlimit --pid "$p" --nofile=524288:524288
done

But it's a band-aid. The problem seems to lie more with MinKnow. It's not closing the files it makes. Let me know what I can do to provide more information or help aid in the quest to discover and fix the problem for good. With files that are not properly closed, there is a real risk of corruption if, e.g. a deflate stream isn't closed properly, etc.

[0x55555555]

Hi @GabeAl,

Can you confirm what arguments you are passing when starting sequencing?

Are you able to run an lsof on the sequencing run while running, providing a list of the files MinKNOW is keeping open?

Thanks,

• George

[0x55555555]

And also, can you confirm exactly what minknow version you are running?

[GabeAl]

Thanks, it is 25.02.12. I wish I'd saved the lsof output previously but when I did run it, it looked like many thousands of fastq.gz files. (Some of these later needed rescue too but not a big deal. Stopping a run because of mishandling them though? Bigger deal). Since I rebasecall anyway using pod5's it doesn't really matter if a few fastqs need a little extra love, but I was also contacted by support telling me a fix for the fastq issue (which may be the same issue) is coming soon. It hadn't shown up last week when I tried updating MinKnow for Ubuntu 24.04, though. Parameters: rapid barcoding kit x96 (sometimes up to 4 running at once). Hac basecaller, split pod5 by barcode, split fastq by barcode, trim barcodes, output every 10 minutes, reserve pores on, all the rest defaults. Cheerio, Gabe

…

On Tue, Aug 12, 2025, 4:05 AM George Pimm ***@***.***> wrote: *0x55555555* left a comment (nanoporetech/minknow_api#80) <#80 (comment)> And also, can you confirm exactly what minknow version you are running? — Reply to this email directly, view it on GitHub <#80 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5NOBQT7EUB7I7PPD5EB7L3NGN6PAVCNFSM6AAAAACDT6XJBSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCNZYGE4TKMJRHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[0x55555555]

Weve been testing this internally and believe the leak of file handles is the same issue fixed in 25.05.14, released last week.

We believe on further inspection the truncation of fastq files was paired with a leak of the handle.

Our runs internally show the issue is resolved when running on the fixed release.

Thanks,

• George

[0x55555555]

MinKNOW 25.05.14 should be available for ubuntu 24.04 - please do let me know if you aren't seeing it.