nanoporetech
Wrong generation of certificates
Good afternoon,
Today I updated MinKnow and noticed that I can no longer create simulated minion with script located in /opt/ont/minknow/bin on my Ubuntu 22.04 Computer.
It was giving such an error
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1749043801.666635 9988 ssl_transport_security.cc:1665] Handshake failed with error SSL_ERROR_SSL: error:0400006b:RSA routines:OPENSSL_internal:BLOCK_TYPE_IS_NOT_01: certificate signature failure
I0000 00:00:1749043801.672114 9991 ssl_transport_security.cc:1665] Handshake failed with error SSL_ERROR_SSL: error:0400006b:RSA routines:OPENSSL_internal:BLOCK_TYPE_IS_NOT_01: certificate signature failure
Unable to connect to MinKNOW at localhost - <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:9502: Ssl handshake failed (TSI_PROTOCOL_FAILURE): SSL_ERROR_SSL: error:0400006b:RSA routines:OPENSSL_internal:BLOCK_TYPE_IS_NOT_01: certificate signature failure"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2025-06-04T16:30:01.672243306+03:00", grpc_status:14, grpc_message:"failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:9502: Ssl handshake failed (TSI_PROTOCOL_FAILURE): SSL_ERROR_SSL: error:0400006b:RSA routines:OPENSSL_internal:BLOCK_TYPE_IS_NOT_01: certificate signature failure"}"
As Windows version works just fine, I tried to dig myself. I uninstalled all MinKnow related files and had clean installation.
I found out that newest installation of MinKnow on Ubuntu 22.04 creates /data/rpc-certs folder in root of file system. I compared ca.crt that were generated in /data/rpc-certs/minknow/ and in /var/lib/minknow/data/rpc-certs/minknow and they differ.
After I removed files inside /data/rpc-certs/minknow/ and put files from /var/lib/minknow/data/rpc-certs/minknow/ script started to work without problems.
I kindly ask you to fix this bug to avoid additional movement of files after installation. Thank you.
Thanks for the post. I have spent a couple of days to troubleshoot the same issue - or even more, and this post definitely gets me a good lead to at least where to look into.
It appears that this originates from an update by ONT attempting to get rid of the internal certificate issue as I discovered two years ago ^1 and occurred one more time earlier this year ^2. This time, instead of shipping their generated certs as part of the package, they setup a systemd unit to automatically generate a new set of certificates upon reboot. The issue is, these new set of certificates were put into /data instead of /var/lib/data, and the user management part is completely misconfigured resulting in the unit always fail.
That shall be an apparent error and one can figure it out immediately as it would result in minknow.service failing to start, but unfortunately another copy of certs (maybe as a fail-safe mechanism?) without the auto regenerate would be generated in /var/lib/minknow/data - by the very service of minknow.service, completely covering the issue up until someone try to use the MinKNOW API.
Apparently the cert autodetection order is also wrong, thus resulting in the minknow.service using the one in /var/lib/data, but the minknow_api python library uses the /data one.
I don't know if this is the right place to post a more detailed bug report maybe with a patch, as the package in question is actually minknow-core-standalone instead of this library, though this package contains this library and something more, and I hope that I am not breaking some EULA kind of thing during this process. Maybe I should follow it up with some post in the closed community instead?
Update: I posted the complete report ^3 onto that page as it contains part of the minknow-core-standalone code.
The fix is to not only remove /data, but also disable and mask the certs-gend.service and certs-gend.timer. Otherwise, the files would return upon reboot.
All following links require an account with ONT.
Hi all,
Thanks for the detailed reports! We have a fix for this issue prepared in the next software release.
If you need assistance working around the issue in the mean time, please reach out.
- George
