Wrong generation of certificates

[Strexas]

Good afternoon,

Today I updated MinKnow and noticed that I can no longer create simulated minion with script located in /opt/ont/minknow/bin on my Ubuntu 22.04 Computer.

It was giving such an error

WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1749043801.666635    9988 ssl_transport_security.cc:1665] Handshake failed with error SSL_ERROR_SSL: error:0400006b:RSA routines:OPENSSL_internal:BLOCK_TYPE_IS_NOT_01: certificate signature failure
I0000 00:00:1749043801.672114    9991 ssl_transport_security.cc:1665] Handshake failed with error SSL_ERROR_SSL: error:0400006b:RSA routines:OPENSSL_internal:BLOCK_TYPE_IS_NOT_01: certificate signature failure
Unable to connect to MinKNOW at localhost - <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:9502: Ssl handshake failed (TSI_PROTOCOL_FAILURE): SSL_ERROR_SSL: error:0400006b:RSA routines:OPENSSL_internal:BLOCK_TYPE_IS_NOT_01: certificate signature failure"
	debug_error_string = "UNKNOWN:Error received from peer  {created_time:"2025-06-04T16:30:01.672243306+03:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:9502: Ssl handshake failed (TSI_PROTOCOL_FAILURE): SSL_ERROR_SSL: error:0400006b:RSA routines:OPENSSL_internal:BLOCK_TYPE_IS_NOT_01: certificate signature failure"}"

As Windows version works just fine, I tried to dig myself. I uninstalled all MinKnow related files and had clean installation.

I found out that newest installation of MinKnow on Ubuntu 22.04 creates /data/rpc-certs folder in root of file system. I compared ca.crt that were generated in /data/rpc-certs/minknow/ and in /var/lib/minknow/data/rpc-certs/minknow and they differ.

After I removed files inside /data/rpc-certs/minknow/ and put files from /var/lib/minknow/data/rpc-certs/minknow/ script started to work without problems.

I kindly ask you to fix this bug to avoid additional movement of files after installation. Thank you.

[StephDC]

Thanks for the post. I have spent a couple of days to troubleshoot the same issue - or even more, and this post definitely gets me a good lead to at least where to look into.

It appears that this originates from an update by ONT attempting to get rid of the internal certificate issue as I discovered two years ago ¹ and occurred one more time earlier this year ². This time, instead of shipping their generated certs as part of the package, they setup a systemd unit to automatically generate a new set of certificates upon reboot. The issue is, these new set of certificates were put into /data instead of /var/lib/data, and the user management part is completely misconfigured resulting in the unit always fail.

That shall be an apparent error and one can figure it out immediately as it would result in minknow.service failing to start, but unfortunately another copy of certs (maybe as a fail-safe mechanism?) without the auto regenerate would be generated in /var/lib/minknow/data - by the very service of minknow.service, completely covering the issue up until someone try to use the MinKNOW API.

Apparently the cert autodetection order is also wrong, thus resulting in the minknow.service using the one in /var/lib/data, but the minknow_api python library uses the /data one.

I don't know if this is the right place to post a more detailed bug report maybe with a patch, as the package in question is actually minknow-core-standalone instead of this library, though this package contains this library and something more, and I hope that I am not breaking some EULA kind of thing during this process. Maybe I should follow it up with some post in the closed community instead?

Update: I posted the complete report ³ onto that page as it contains part of the minknow-core-standalone code.

The fix is to not only remove /data, but also disable and mask the certs-gend.service and certs-gend.timer. Otherwise, the files would return upon reboot.

All following links require an account with ONT.

Footnotes

1. https://community.nanoporetech.com/posts/minknow-grpc-certificate-e ↩

2. https://community.nanoporetech.com/posts/certificate-expiry-for-min-15102 ↩

3. https://community.nanoporetech.com/posts/minknow-core-v6-4-9-comes ↩

[0x55555555]

Hi all,

Thanks for the detailed reports! We have a fix for this issue prepared in the next software release.

If you need assistance working around the issue in the mean time, please reach out.

• George

[StephDC]

Following up with this issue.

I have upgraded to minknow-core-standalone 6.8.11 available Oct 29.
It now comes with the hard-coded /var/lib/minknow/data as the output directory, thus fixing the /data issue and the failed to connect issue due to using different set of certs. Thanks for the hard work.

However, the L9 chown mooneye still persists. The certs-gend.service would therefore start to generate the certificates, then fail due to referencing the non-existing mooneye user.

I would like two clarifications:

1. Does the mooneye user suppose to exist on all machines installed with minknow-core-standalone?
   1. if so, should we manually create it? As unlike the minknow user created by minknow-core-standalone preinst, it would not be created automatically.
   2. if not so, should you mark that line as optional or instead of a later chown call, handle it in rpc_certs_gen program?
2. Do you have any plan to remove the /data/rpc-certs created by previous bad certs-gend or should it be announced somewhere?

[0x55555555]

Hi @StephDC,

I'll work with the internal teams to further address these issues.

For now, if you have badly generated certificates can you manually remove them to get running?

Thanks,

• George

[StephDC]

Hi @0x55555555 ,

I have programmed my tool to detect the presence of /data/rpc-certs to prompt the user to manually remove them.
I also drafted a notice for my tool's users to look for that and remove them in case my tool did not function as expected.

Overall removing that directory did indeed solve the problem.

The thing is, removing /data/rpc-certs usually means privilege escalation to root (I believe no one would like a / with g+w or o+w). My tool also runs on other computers that I do not have access to, so requesting root and automatically removing something out of / not created by my tool also sounds a little bit out of reach. That is why I preferably would like something from ONT to remove that bad directory, or at least instruct the user to do so.

Nonetheless thanks for following up and at least on my side this is no longer a critical bug but a mere annoyance, at this version.